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COMPTROlipR GENERAL OF THE UNITED STATeM 
. WASHINGTON, D.C. XOMS 



B-183442 

The Honorable Olin'^E. Teague. L . . * . 

Chairman, Committee on Science and 

Technology ' • - 

House of Represent^atives 

' <. .. ' ' .} 

Dear^Mr. Chairman: / ■ ' • • 

In accordance wi/h your March 13, 1975, request, we 
have reviewed (1) the? development, evaluation, and implemen- 
tation of the National Science Foundation-supported science 
education project ^'Man: A. Course, of "Study" and (2) the rela- 
tionships between the Foundation and <^the project's developer - 
(Education* Development Center, Inc.) and piAlisher (Curriculum 
Development Associates, Inc.). As -agreed, we obtairted the 
views OL the Foundation and the » two private firms on .our find- 
ings and- their' cbmments are considered in' the report. 

As a result of our f iadings, we are making a number of 
recommendations to the Foundation's Director to improve the 
aeneral administration of precollege science education ac^ 
tivitj.es. In addition, we are recommending that the Chairman, 
National Endowment for the Humanities, review the Endowment's ^ 
authority for a questionable grant it made to the Education De-- 
- velopment Center and deter^mine if the funds should be recovered. 

We believe the contents of this- report would be of inter''-, 
est to committees and to other' Members of Congress. As- you 
know, section 236 of the' Legislative Reorganization Act of 
1970 requires the head of a Federal agency t9 submit a written" 
statement on actions taken on our recommendations to the House 
and Senate Committees or^ .Government Operations not later than ^ 
*• 60 days after the date -of the report,, and to the House and • 
Senate Committees on Appropriations with the agency's first 
request for appropr iatio'ns made more than 60 days after the 
date of the report. .We will be in touch with your office in 
the near future to. arrange for distribution of the report to - 
the agencies involved and to-the four Committees to set in 
motion the requirements of section 236. , , 

Sincerely yours^ 

• ' • %^ 't^- A^^ . 

" ^ - . < Comptrolfer General 

of "the United States 

ErJc • 00003 
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COMPTROLLER rGENERAL'S REPORT ADMINISTRATION OF THE 

TO THE HOUSE COMMITTEE SCIENCE EDUCATION PROJECT 

ON SCIENCE AND TECHNOLOGY • "MAN: A COURSE OF STUDY" 

(MACOS) 



Rational Science Foundation 



DIGEST 



"Man: A Course of Study," published in 1970, 
is a social studies course (generally for 
grade five) developed with National Science 
Foundation support totaling over $7.4 mil- 
lion., 

GAO's study of that project suggests that 
administration of precollege- curriculum proj-' 
ects could be improved to insure that sound 
business pfacricesT^are^f ollowed rby'-Foundation- 
officials and recipients of projgct funds. 

Some projects produce income, such as. royal- 
ties, which the Foundation may ^^author ize 
project grantees or contr actprs ^to use , but 
'such earnings and^ use are not repprted to 
the Congress. The Congress may wish the 
Foundation to' determine the significance of 
such income, and require a report of re- 
ceipts and expenditures for use in consider- 
ing the Foundation's annual appropriation 
request. (See pp. 44 and 48.) ^ 



' GAO recommendations'^ to strengthen management 
of • Foundation projects include: . 

— Ascertaining if a competitive process is 
feasible and. -effective for developing edu- 
cational products. ('See pp. 4 and 10.) 

--Establishing procedures so that files are 
documented to show (1) disposition of 
project evaluators'* comments and (2) rea- 
sons for supporting or riot supporting proj- 
^ect- implementation proposals. (See pp. 8, ^ ^ 
10,\39, and 42.) 

— Requiring, documentation, to suppo^rt reasons 
given by grantees or contractors in select- ^ 
ing a publisher to market educatipn mate- 
X Dials developed with Foundation support and 
^ insuring that publisher interest in mar- " 
keting' such materials is 'redetermined when* ^ 

Ttar Shf t . Upon removal, the report . 

J^tr^a|e should be noted hereon. • 1 MWD-76-ZO 
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conditions change that could affect 
.publisher selection.' (See pp. 22 and 30,) * 

—Reviewing all 'contractus and subcontracts 
for ^maiTketihg^ducational materials, (-See 
pp. 25, and 30. ) " , . 

— Conducting a review of the "Man: A Course* 
of Study", developer 's royalty fund tp-anp- 
actions, evaluating the review results, in 
considering the need fdr ifiU^re frequent 
audits of the royalty fund, and more 

• closely examining that developer's income 
reports to help monitor the royalty iEund. 
. (See pp'. 44, 46, and" 48..) . o 

* * ' 

The Foundation ha s no guidelines for eval uat- - 
ing tne effectiveness oJE educational "produQts 
it supports or formal procedures for deter- 
mining^ any adverse , impact on human subjects 
using »the inaterials. (See pp. 11 and'150 

The Foundation funded^ an extens^'ive evaluation 
of the ^project before^ commercial di&J:ribu- 
tion; 'however", evaluation ^claims of signifi- 
cant learning gains for children taking the 
course were .questionable because of limita- 
^tions in the evaluation des^ig^ri. , (See p. 11.') 

GAO in another' study is considering evalua- 
tion, of Faundation-suppor ted education proj- 
ects but in -the winter im recommends that the 
Foundation's Director': ^ • « 

— ^^Establish procedures/ for selecting peer re- 
viewers off proposals for curriculum develr- 
opment to insure that views of intended 
users, such as school administr atprs and^ , 
teachers, are obtained., (See p. 10.) 

t.~Review the *need for establishing procedures 
' to safeguard human subjects involved in its 
educational activities. (See p; 17.) 

The Foundation agreed with GAO.'s recommenda- • 
tions. (See pp. 10, 17, 31, 42, 48,^ aiid 55.) 



GAO identified one questionable transaction , 
between the Foundation and the project der 
veloper and another 'involving . the. developer ^ 
the publisher, and the National Endowment 
, for the Humanities. GAO recommends- re\!i«w - 
of the transactions and necessary adjustments ^ 
(See pp. 27, 31, 46, ah^ 48.) ' , 
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<y CHAPTER 1 

] INTRODUCTION 



."\ Pursuant to a March 1-3,^19,75, req 
Houfee Conmiittee on Science and Technol 




St of the Chairman^ 

and subsequent 
aspects of the 



House v^omm^.u'-cc ^^v.*^ ♦ . . ,i 

slleicriSu^daUon-s^PPorted 'science cjriculum development 
Iriject -Man: .A-'Course of Study" (MACQ^. We considered 
the: • . ' ., \. 

— Foundation '.s policies aind pr6cedure\f or (1) develop- 
. ^ing/ evaluating, and 'impl erne nti^ng s?^^"^^ ^"^""^^^^ 
•prajects at the precollege level and for (2) dis- 
posing of royalty income- from the projects. 

—Extent of boiiTpetition in\^olved in -the, envelopment 
^nd, marketing of MAC05< 



' • —Propriety of a professional ser vicey ap^em^nt 
between the MACOS developer and publisher. 

—Determination of the rate for royalty payment to the 
Foundation resulting from HACOS sales ana the dis- 
Dosition of MACOS royalties due the Government, and 
gene^aUy examined the MACOS developer^ s royalty poo 

~ used to account -for royalties received from all 
Foundation-supported science educatioTi projects. A 
financial audit was not performed due to time con- 
straints. * * 

" --Foundation's policy concerning endorsing science 
education materials developed with its support, 

—Evaluations of MACOS during development. 

—Foundation's policy and. procedures for protecting 
' • . human, subjects .involved in its science education--^ 
activities. 

* — kckground data on the MACOS developer and publisher 
. ' and -their relationships *ta the Foundation. 

« ^ • * 

'MACOS is a Foundatiori-suppd'rted, social studies course 
(geniraliy for grade five) developed by the. Education" 
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Development Centery Inc. (EDC), 1/ a -nonprofit^ corporation 
engaged- in educational research arid. aevelapment lofcated*in* 
Massachusetts.- The MACOS, mater ials were published In "1970 
and are being (iisseminated under contrac-fe "with *BDC by Cur- * " 
riculum Development ffssociate>> Inc. (CDA), a commercial , 
corporation fqr developing and publishing education cur- 
rviculiims^ located in-Washington, -D.C. ' o 

MACOS uses Stttdies of selected animal groups and^ the ' 
l^fdtsrlik Eskimos—a simple human society — to explore* the 
roots ofjiiiman social * behavior^. According tj> EDC, MACOS 
emphasizes^ the bioloaicai continuity frdm- animals to hCimans ' 
and the distinctiveness and diversity of the human cultural 
heritage* Course %ate'rials ^include f ilmsr ^.fj-lmstr ips, > »^ 
\slid^s, records,, booklets, chaDts / games, and displays • 
CDA estimated tha^t MACOS materials were being used in aboujt 
lV700 schools in 47 States. 

The "development of MACOS was , supported under the National 
Science Foundation Act of 1950 (42 U.S.C. 1861 et 'seq % ) , 
as amended, which authorl^sed the l?oundaticn to initiate arid 
support (1) basic scientific research ajid ^(2) programs 
to increase research t>otential through contracts or other* 
f6rms ^f a-ssistance,* such as gr-ants. • ^ '\ 

* « • 

The Foundation's science education activities, 
admin is tere(J by its science'.education directorate, Qonsist 
primarily of grant and fellowship progran\s intended to im- 
prove education for professional careers in science- and 
technology-based" fields, improve scientific literacy, and 

.increase* the efficiency and effectiveness' of educational ^ 
processes. Over the last 10 years (fispal^ears 1966-75), 

, these -activities have received over $1 billion in funding. 

* The Fecundation identified 53. pr.ecollege-level cur- 
riculum projects — including. MACOS--undec .the science educaT^: 
tion activities. for which.it provided about. $196 million \ 
in funds during fiscal years U956-75. EDC received about 
*?4.8 miXlio^n dur ing ' fiscal years 1^63-70 for developing a 
social studies curriculum program lihat eventually evolved 
into MACOS.' VarTpu's grantees received ^ over $2.3 million* 
•for implenrenting the materials during fiscal years 1967-75 
and about $326,000 for postevaluation of MAc:OS between 
1970, when it was published, and June 30, 1975. 



I/Formerly Educati^onal Services Incorporated-, 



ERiC • . 2 

. 'Vj^nfu^ . 00010 



. The science" education directorate, headed by the 
AsSi'startt Director for Science Education ^ consists of the 
' following ^units-* * - • : ' 

'.^ 1/ 'Office of Expei:imental Projects and Programs — ' 
c<yhc>erned with discovering new/methods f or im- 
prpving science education and \he testing \and' 
> / evaluation of' the- methods- and with developing ' 
and administer infci highly experimental aQtivities. 

2. pivision of fr.e-College Education in Science — 
responsible for developing -and /implementing, m^eans ^ 
•to improve scierice instruction for kindergarten 

• ^ through the l-2th grade/ raising th^ .Nation's . . 
science literacy level,* and improving the in- - 
'StructionaX process/ ' * • v 

3. Division of Higher Education — generally, the same 
as precollege division except that i»t is, con- 
cerned with post-secondary academic levels and 
continuing ^ducation^ programs for career i 
scientists and engineers* 

The MAGOS project is VdministeredAy the precollege 
division/ which contains a mater ials-and instruction 
development sectioi^and an instructional improvement im- 
plementation section* Wiiihin each section program managers 
are responsible for administering project grants and con- 
tracts* . We primarily reviewed the policies^ procedures, 
Snd practices at the precollege level *for supporting the 
development, evaluatign, and ^implementation of science 
education proj'ects and the application of these policies, 
• procedures, and practices to KACOS* * 
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CHAPTER 2 . 
DEVELOPMENT OF MACOS 



Foundation off icialv<5 identify broad educational needs 
and express these 'needs to the educatAon and scientific com-' 
raunities through {>rograin announcements. In response to • 
these announcements, unsolicited proposals for projects 
to meet these, needs are submitted to the Foundation. The 
p^posals for product development are received throughout 
the. year and generally do not compete with other proposals. 

Product development proposals are usually sent out- 
'"side the' Foundation to peer reviewers, who are requested 
to review them and comment on their merits. For thie- MACpS 
program/ however /. Found^ition officials could not furnish 
us witli "documentation showing the disposition of many o£ , ^ 
the peer reviewers' comments. ^ Also, most peer reviewers 
were affiliated with colleges and \universities and few 
(3 out of 35) were poteAbial users \of the materials, such as 
elementary school administrators and teachers. 

PROCEDURES FOR AWARbiNG PRECOLLEGE • 
SCIENCE BDUCATIUtj DEVE:.0PMENT PROJECTS 

No formal procedures exist for assessing educational 
needs. Instead^ Foundation officials identify broad needs 
thr'ough informal means, such as conferences, Auvloory Com-^ 
,^ittee 1/ meetings, ideas from experts in the ^ield, and in- 
ternal exper'tise; through proposals received; and through . 
a Variety, of projects focused on problem and need.s assess- 
ment. The-broad needs are subsequently expressed to the 
education. and scientific communities by a number of mech- 
anisms, - including professional journal articles^ books, 
and projject reports or by special announcements and pro- 
gram announcements by the Foundation. 

The program announcements for the development of 
educational materials are circulated to individuals that 



1/The Advisory Committee for Science Education is to pro- 
~ vide advice and recommendations concerning education 
.activities to the Director of the Foundation, through 
the-Assistant Director for Science Education. The Com- 
mittee consists of froi^ 9 to 12 iTiembers, who are appointed 
by the Director for 1-year terms and are normally not 
reappointed for Tnore than 3 consecutive terms. 
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: request .them ".each yeaf . According to a Foundat.ion otxi^ 
cial, this, is- done to. k^ep the number of proposals, re- . 
.•; ^ceived to manag.eable Propojrtiong.. Approximately- 2,000 
program annoujicements for ^ de^^elopment of educationj.1. 
-materials- are distributed yearly-, -Ho.weyer, other Founda- 
tion publications th'at describe the .science education 
•activities and reference the ^documents that interested . 
.■7 persons, may request are much more widely distributed. 

• The announcements are included in the FounxJation ' s 
^uide for -preparing propp-sals and opeirat.ing proDects for 
, ' materials and instruction deyelopment. The announcement 
^ ^ guide defines the eligible academic grade levels, organi- . 
zations, and^eMs for, which the -Foundation, will accept 
■* proposals.- 'tEpPfactQt:s halve remained basica^^ly un- 
" changed over ^th^ Nyears.'^ ^ ^. ■ . \ 

' " ' ■ ' ' • ■* - \ ■ • ^• 

. in the materials and instruction .dev.elopment\gui.de - 

for fisbal year 197*5 proposal^., ( 1) .pr*oj-ects werg^;i;o 
cover grades, kindergarten- through 12, (2) the eligible 
organizations generally included colleges and univer siti-<5 
on behalf of their staff members .and nonprofit organi za- 
*• -tions; such as professional, scientific, and educational 
associations or ' societies ; research institutes and lab- 
oratories; and education coAsortia, and (3) the eligible, 
• fields included mathematical,, physical, biological, med- 
ical,, engineering, Vnd social sciences and the/istoty 
and -philosophy of science. • " - ■ ' 

" - ThV announcement guide also, outlines the broad cate- 

• , gories the Fpundation J/ill consider supporting 

■ 'ing project'^. These categories have changed as educational 
ields have .changed.' Some categories included m the fiscal 
*. year- 1975/gtiide were: ^ \ , * 

—Commi'tjtee and conference 'studies designed to 

identify Prcrbiems in a given field and. to forni- 
" ulate guidelines for the evolution of modern m- 
t structional programs. ^ • 

Project^ 'to-; develop models of outside-the- 

. * classr<?bjTV instruction. , ' ^ 

Projects to develop modes of instruction 

realistically approaching the needs of non- 
V • academically-oriented students.' * . 

Projects to develop model courses or course 

sequences using many types of learning and 
teaching aids. 



The Foundation r-ecogrfized- that these were not the only ac- 
tivities worthy of support, but merely examples of broad^ 
c$itegories of' activities 'fo.r which supper t was appro- 
priate. Also" noted in the. fi,sc^al year 1975 guide were 
broad area^, such as sSeveldping materials relating sci- 
ence and tec-hnology to environmental and societal prob- 
lems, to which- the Foundation would give high-puiority ^- 
considerat.ion. . • . ' . 

The precollege material's ,and instruction development 
guide is a public indication of'the Foun.dation' s areas 
of interest, but the Foundation . does no,t .consider it a 
solicitation.. As a result, pfoposaTLs 'r?.ceiv.ed. by" the 
Foundation for- developing. educational materials aire gen- 
erally unsolicited. There iS no deadline for the fec^ipt 
of these proposals,- which are received throughout t^he 
year and generally do not compete with other proposals. 
According to. a Fouhda.tiqh official, the Science education 
dirjeotota<te accepts unsolipited proposals because it pre^ 
iers to i,<aentify. broad needs and rely on scientific and 
educational experts to. propose specific ways of meeting 
those needs. This also, allows proposers mbre freedom _ 
to deve^lbp their own ideas. • " , ^ - 

When a preliminary proposal for developing educatidnai 
materials is received, the Foundation determines whether 
the proposal is 1&otehtially supportable. This initial de- 
termination is generally made by the Foundation's, program 
Managers. If the proposal lacks- some oasic required 'in- 
formation (such as budget data and objectives), it is r,e- 
turned to the proposer for completion; "if it does not 
meet all the basic requirements (such as elig ible. organi- 
zations and -fields) , the ptovoset is discouraged f com " 
submitting a formal proposal. 

* 

Once the proposal is determined by the Found-ation 
as being potentially supportable, it i3 usually suoj.ect 
to a mail review in which' outside peer, reviewers are 
requested to examine it and comment on its merits. 
The Foundation usually also sends the reviewers a let- 
ter noting the factors to be considered in reviewing 
the proposals* Such factors include scientific merit, 
the feasibility and effectiveness of the proposed ,pro- 
«<2tedures, competency of the investi'gator s , reasonableness 
ot the budget, potential for community impact, and com.- ^ 
mitment of non-Foundatio'n sources as sporiiSors*^ ' The out- 
side peer reviewens are primarily scientists a^'nd educa- 
tor s . competent in the fields-, involved in the proposals. 

upon receiving the reviewers' comments the Founda- 
ttion's program staff assesses them and_ their impact on 

000.14 



3 



Gen- 



a pt^oposal and may negotiate ^oth substantive and fi- 
nannilv a«!Dects of the proPOsal wfth the proposer. 
2?2nv a summarrof pertinent re^>,iewer.s comments is pro- 
^videf to the p?opose/for his, consideration. Whea the 
proposer . and foundation atatf . reacrt'agreement , a. recommea- 
da^ion for support is prepared. The Foundation makes the 
f?naa decision as to ^whether ^ propa^sal- will, b6 funded. . 

* - -Proposals that -the Founda-tion's *prpgram sta^f^haye ; ^ 
determined will not be .supported, may bei withdrawn, by the ^ • 
■proposer! The Foundation provides the opportunity to 
withdraw based mainly the proposers assumption^th^^ . 

a 'denial could be detrime;ital to them ^« .^"^'"i"'"?' the 
ture proposals. Defiled proposals are signed of-f at the ... 
division le^,el a?ter being" reviewed ana- asse.ss,ed by the . 
.^^^^grrm'ni^gersWd. section heaB the- precollege 
materials^a- inst-ruction, development section. 

y Approvals of s-upport. require the signature of fehe. ^ 
.Assistant Director for Science Education after the pro- ^ 
g'aiianagers, ±he section 'head, and -.the ^1^^=^°" ^^^^^^^'^ ' 
hive reviewed and . commented on the; proposals. Awards 
. involving expenditures 'of at least. $50.0 , 000 in a single- . 
ieir or ae least $2,000,000 In total must also be ap- 
proved by the- National Science Board. Projects are' 
generally funded annually, and renewals requesting -ad- 
ditional support Ire also us.ually .subject to peer re- 
view. * ' ^ . ■ - " ' • 
HISTORY OF- .MACOS., DEVELOPMENT " ■ ; 

'oiirinq' 1963^69, the Foundation awarded EDC three grants. - - 
tor the-'de^eloi^u^nt^of^a Special* Studies Curriculum Program. 
T^l of 'the gra^, totaling about .$195 ^900 , w>ere for 
producing- a. series of- anthropological, films. The third 
q^ant^'wbich was amended 14J:imes and -feotalea $4.6 million, 
- included working" further on the -films asnd developing cur.- 
. rScuJum materials\f^r elementary and hig.h school s.egments. . 
"of the ..program. ..MACOS evdlved from the program's elementary 
segxrtent.' , . , ^ - • ' > " 

vThe sFaisi 'studies Curriculum Program was conceived 

■ at av.igfr2 conference sponspred by tEUC, the American Coi^n- . 
•'cil'-oi Learn^dVsbcietles,- and the Ford a'nd Sloan .Founda- 

• t ion's ..l-feonference participant? noted that JiLUch material 

■ taugh^in -social studie-s &nd the 'humaiiities- was primarily 
histor/; and-'they exploVed po*ssibiUties_ for ai, substantial 
cirrt^Ium revY'sidrf in these areas. The program was antended-- 
to produce an integrated cur r iculum in social stqdies and ^ 

the humanitie.s 'ftfr grades ..kindergarten, through,12.^^ 
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Under ^ the three Foundation grants awarded in 1963, 
EDC waa to produce a sferies of anthropological f.ilms that 
were to form a central part of six social studies units for 
grades one to' six. In 1965, EDC received additional fund- 
ing for developing curriculum units for .other educational 
levels. , < . 

c * 

By the" beginning of 1966, the elementary segment 
- evolved into MACOS. A junipr high segment, not funded " 
/6y the Foundation, was , to cover Gree.k and.Roman civiiliza-- 
tion and to contrast 18th-19th egntury England and^Amer- 
ica. The work at the senior high level was eventually 
limited to developing a lOth grade course on the impact 
of technology and science in 19th century England. The 
• Foundation discontinued' support for that course in 1968 

because .the developer had trouble ^developing the^r project 
' materials. 

' PEER/ EVALUATION OF EDt'S . / / 

" PROPOSALS FOR -FUNDS . • ' ' 

[in May 1963, EDC submitted-to the Foundation a pr'or- 
posai £hat ^consolidated four earlier proposals seeking 
support of film making for its elementary school social ^ 
science program. Foundation ^official^* obtained peer re~ 
<w view comments on the earlier film prbposaJis but not on 
the consolidated propasal, because most 6f the infor- 
mation 'included in the latter proposal had also been • / , 
presented in the earlier ones. EDC subjnitted five * 
more proposals to the Fou'ndation- as amendments to the * * 
consolidated ^proposal ; peer review comments were not 
.obtained, on two*of them. , One authorized EDC to make 
course mafcer ials. available to schools; in thi^s instance. 
Foundation ofcfi^^als believed a peer review wa? unneces- . 
sary. The other^involved teacher training and evaluation; 
Foundation official's could 'not explain -why peer review 
comments w^re nqt obtained in this instance. 

^ _ ♦ » ♦ 

•According to a Foundation official, the Fotindation 
considers all reviewer comments- and communicates them to . 
the grantee either* in a written summa'ry or by telephone. 
•Cancerning EDC* s . project , most reviewers recommended ^ 
project ^approval ; several canstr udtive ciritiaisms con- 
cerned course content and^the evaluation *plan. For ex- 
ample, thr^^ reviewers^ commented as follows. 

, * 1. "The Eskimo film is also ^excellent. But the ^ 
problems of using the'Eskimo way of life in 
* * * grammer school has not been faced. If a unit 
; actually iridicated how Eskimos lived including 
! promiscuity and cannibalism, no srchooJL board 
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in the country^ would allow the material to be 
presented." 

2. "I am less impressed with the expected -outcomes ^ 
of such jiew curricula. What impact will these 
courses have on 5th and ioth grades other «thao ^ ^ 
the imparting of information on baboons'^ Eski-^ 
m'o^, and Darwin? How will students be differ- 
ent and better for having been exposed to this 
material?"- , 

— - 3. "The rationale of the evaluation, program * * * 
does not satisfy, especially because it does 
not specify evaluation in tt\e only te^ms that 
make educational sense—in comparison to some- 
thing." ' ' 

Foundation records did not indicate, and -the program 
manager did not know, what disposition was made of these and 
othe'r comments. 

Mix of peer reviewers ^ '^^^^ 

(, 

In. total, 35 peer reviewers were selected by the Founda- 
tion to comment on the proposals which evolved into the MACOS 
project. Most of the reviewers, who 'were from the disciplines 
of anthropology,, education, sociology, social' psychology, and 
economics, were affiliated yith coll'eges and uhivc-r sities . 
Of the 35, 3 w^re from potential user groups, such as State 
educatioji agencies^.and„eA,ementa]Ly^^and-^secondar-y— s /- ^ 

CONCLUSIONS ' , , ^ ^ 

The Foundation generally receives unsolicited proposals 
for developing educational matetials that do ,not compete 
with other proposals. As much competition aS practicable 
should be obtained to promote efficient and economical . pro- 
curement. The Foundation should review the science educa- 
tion needs to identify specific areas that lend themselves 
to ,coinpe%itive procedures, such as formal^ re.quests for pro- 
posals. 

The Founda^on's program manager for MACOS advised us 
that all peer review comments were considered; thus, we 
cannot say that the project would have been any diffe'-ent 
had 'the comments been formally considered. However", b^ 
cause 'peer 'eValuation is ,an .important part of the. Founda- 
tion's grant evaluation procedure, we believe^that the , 
extent of consideration and ultimate disposition of the 
peer comments "^should be documented. The Foundation, in ^ ./ 




obtaining peer evaluation of proposed educational materials, 
should especially obtain the views of the intended users in 
the educational ^community. \ ^ ' ^ 

RECOMMENDATIONS • . ' . • 

We tecommend that the Director of the' Foundation: 

— Experiment with using competitive devices » such as 
formal requests for propo8al.8» to ascertain, if a 
competitive process is feasible' and effective for 
developing educational products. . - " ^ 

— Establish proceduifes to insure that the evaluation 
and disposition of peer review comments are documented 
in the, project files. • 

» T-.-Establish procedures for selecting peer reviewers to 
insure thvit the views of intended users » such as 
-school administrators and" teachers, are obtainedV 

AGENCY COMMENTS \ 

By letter dated September 15, 1975 » the.Poundatioh 
' agreed' with our recommendations and stated tthat actions needed 
to implement them are being taken. (See app. II.) 




CHAPTEk 3 



EVALUATION OF MACOS 



EvaluaTipTrTB^h-e-^^ the Foundation, the 

project group,, and 'the educator can determine ^whether the 
, completed project has successfully achieved its objective — 
improving science education. The Foundation has no formal 
guidelines specifying the types of testing and evaluations 
to "be performed on educational materials developed with its 
support. ^ ' , ^. 

J , EDC comprehensively evaliua'ted MACOS. Based upon its* 
tests.,...EDC claiitied -that students^^art iclp'at ing"^ in' i"Es* evalua-^ 
tion program achieved significant learning gainst . However , 
the significance of pre-*to pdst-test gains* attributable to ' 
MACOS was questionable because ^f limitations .in EDC's evai:ua- 

'tion design.^ Fur thenmor-e, a xeductlon in the sample that 
could have greatly affected tesi outcomes was not alde<3|uately ' 
explained in EDO's evaluation report. 

The Foundatiqn has adopted the Department/ of Health, 
Education, and Welfare's polity for protecting the rights and 
welfare of human subjects involved in its supported activi-^ 
•ties, but it do§s not apply the policy to^ its educational 
activities. / 

TYBES OF TESTING. AND EVALUATIONS OF MACOS • • ^ 

During j:he initial phases of MACOS developmen^v* EDC pro-,^ 
po^ed and conducted' evaluatipns. Using field test results, 
these evaluations wjbre primarily designed to provide a basis 
for revising course .mater iSls. In school year 1965-66, EDC 
recorded student and EDC observer reactions to course materials 
iji local classrooms in which MAGOS was taught by EDC ^teacher s . 
"^In'Hhe following school year, 20 teacher^ not" associated with 
p.roject development used MACOS in their classrooms, and the 
materials and sequence of presentation yere. evaluated and 
ab^oUt 50 students were interviewed to determine their reactions^ 
to the course. Based upon these ^tudigs, the cours^e rqatecials 
were modified. ^ 

\ . ' , ^ 

Further extensive, testing of MACOS for revision purposes ' 
was conducted during 1967-69. The evaluation consisted of 
interviews with students and teachers, observation of class- 
room, act ivities , objective check lists^of classroom environ- 
ments, and objective test'G of content and concept learning. 
Objective data was statistically analyzed and. subjective 
data clinically analyzed. Over 3, 000 students from 1^62 urban 
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and suburban classrooms' 1/ across the country were included 
-in the. .e\?aluationr Pr^r- and po«t-tests containing multiple 
choice and open-ended items on information concepts , a,nd 
attitudes were ^administered* In. 19 classrooms, 137 ;students 
■and their teachers were selected for interviews to determine 
their reactions to the courise. . ' 

'■ ' " ' 

In addition, EDC staff made^ 109 observations in 29 class- 
rooms to^' see the course in action and to evaluate teaching, 
styles. Teachers evaluated the course through questionnaires; 
"70 were returned.' In 14 classrooms from 5 school systems, . 
350 non-MACOS students and 4 of their teachers were observ.eq 
- and interviewed; however, this was done only for 1968-69 • 
'was not .used. to support the learning gain claims- demonstrated - 
by pre-t.est, post-test results.. . • * ' 

■ SIGNIFICANCE OF LEARNING GAINS ^„ 

. MACOS has two major' sections, Man'and Other Animals and/ 
the Netsilik unit'. EDC claimed that,- based on its studies of 
■participating students, children taking both sections of 'the 
" cdiirse made 'significant learning gains. .According to a 

proieat evaluator, the same students «Weffe involved m the - 
pre- and post-testing (using the same test-) for the curriculum 
evaluations. In this connection, among EDO's major findings 
on Man and 0 thee -Animals were- that (1) learning gains were 
not associated with the students' intelligence or previous 
knowledge in the area" and (2)- those s,tudents with pbor / 
oacademic background ,. found so often in the iViner city, gained 
in learning and mastery over the . ideas and concepts as much 
as .those whose beginning positions -were- much stronger; These 
claims were reiterated by CDA, the publisher for MACOS, m 
its brochure advertising the course. Concerning the 
Netsilik unit, EDO qualified its claim for students with 
poor academic background, concluding . that test- gains were 
greatest for those students of highest intelligence quoti,ent. 
and least for those of Iqwest. v ■ „* 

Evaluation experts state that, when a one-group, pre- 
test, post-t<est design, is used, resulting changes cannot be 
Iscribed to the program being evalua.ted. Thus, attributing 
significant gains*. to the project curriculum is questionable 
bec^aise the maturing of the student, differences among " 
t'ea^rhers, and the "testing" effect (.that is, students taking 
a test, for a second time usually do better than those taking 
it forHthe first time) may have contr ibuted , to the g.aijis. 



1/Ihcluded 14 school sy.stems in 1967-68 and 6 in 196J-j69. 
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I.n acknowledging such factoxs, project evaluators admittef - 
that they Kad npt determined- the. impact of such effects on 
tlieir evaluation. . . " ' . ^ ' 

Furthermore, a test' standard--that is, an expected level 
6f achievement-had not been, established before the field 
tests. Without such a- standard, measuring the relative im- . 
portance of .par ticipants' gains is difficult. As one alter- 
native-, a comparison with other students or classes using 
other social studies curr.iculums (and related standardized 
tests which were available) could have partially measured 
the .significance of gains. 

'project' evaiuatdrs .stated that, . despite ' such factors as 
the absence of^ a control sample, in their ^uagment statisti- 
cally significant learning gains were related to the MACOS 
program. However, because pf, the. lack- of evaluation data-, 
tested learning gains realized by. the participating students 

•cannot be shown to--be the result of the MACOS program. We 
believe the statistical significance of the . learning g.ains 
was obvious. For example, our analysis showed. that (because 
o;f the high numbers of students tested)*, for a statistically 

. Significant learning gain to have occurred, students ^would 
have had to increase their pre-test scores only by less, 
than, one more duestion answered correctly (0.57) the ^ 
cost-test, or from 34.4 correct answers (see the following 
table) to about 35. This strengthens ouc belief;- that ^ 
pre-establishid test standard or objective was neede.d to 
measure the relative importance. of the results of the MACO^ 
testing- " ■ » 

EDC, after testing th^ Man and Other Animals unit, 
stated that students with pooj: academic backgrounds, •. 
found so" often in the inner city, gained ap much learning 
and mastery over the ideas a^d concepts as those whose _ 
beginning cositivons //ere much stronger. However, EDc aia 

-not demonstrate that inner city stud.en.ts gained as much 
as students from .other systems. Our analysis of availaole 
EDC summary data, showed that, while inner city stuaents aia 
q'a-ln-,- their gains were less th^n the gains of suburban ana, 

■ other uroan students. This analysis is presented in the 
'fpllowing' table. 
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^Results (mean) 








Pre-test 


'Post-test - • 


Type of 
school 


Number of 
students ,t 


Number 
of itejns 


cpr rect 
answers 


, cor rect 
answers ' Gain^ 


Inner city 
* Suburban and 
d other urban 


439 
1,207 


65 
65 


31,1 
35.7 


39.8,. 8.7 


Total 


1,646 


65 


34.4 


44..4 10.0* 


eOMPLfiTENESS*<- 


* 
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' in 'evaluation report,". EDC claimed that, for test 
year 19-67.-68, 2,182 students participated in • the field test 
of both sections of the- MACOS materials.' However, results 
were reported for only 1,646 students foi? the Man and Other 
Animals, segment^ and 782 students for the >j^tsilik unit. Among 
reasons cited by project evaluatqrs for .the differences in 
■ numbers was the" fact that participating students were" unable 
to complete 'both sections of the course in -one school year. 
We believe that this fact., and the other reasons for Ahe 
differences, should have been adequately reported (through 
the evaluation report) to educators considering purchasing^, 
the product. The importance of such data was shown by EDC"s 
statement that its materials could not succeed without the - 
N^'tsilik segment and that teachers should make sure that the 
segment received its full share of time during the school year 

FOLLOWUP EVALUATION * . 

In ,1970, the Foundation "awarded the Washington School 
of Psychiatry $14,000 to undertake a feasibility study to 
identify evaluation approaches for MACOS. Antioch College . 
was awarded -$49,400 in' fiscal year 1972 fend $262,600 in 
fiscal year 1974 to make a comparative followup evaluation*' 
of MACOS resulting frwom the feasibility study. ^ The evalua- 
tion will include 'a standardized social studies test; a 
ore- and post-test, to be given to both MAG03 and non-MACOS 
students, that will include some MACOS. and non-MACOS speci- 
fics; ^nd classroom observations with followup interviews. 
The evaluation— scheduled for completion in January 1977, 
'approximately 6 years after MACOS ouol ication~is *in-\ 
tended to orovide answers to the , following questions:' 

— What do students who take i4AC03 learn? 

—What do they retain? ^ 

— Is what 'MACOS students learn different from what' ' 
non-HACOS stuJents learn? 

14 
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PROTECTING- HUMAN SUBJECTS ' - 

•■ ; ^ ■ ' . ■ ■ 

■In December 1971, the Department of Heal,th, Education, . 
and Welfare issued "The instituti'ohal Guide to DHEW Policy 
on Protection of Human Subjects" to safeguard the rights ^ 
and welfare of human subjects involved in aativites supported « 
by its grants or contracts. Persons who may 'need- protecting 
includ,ed ^Jatients, donor's of organs, and students in educa- ^ 
tiorial training* supported by the Department^ The injuries 
to be^protected against were broadly defined"^ physical, 
psychological,^ sociological', and others. The institution, 
requesting Department support* was expected to apply the 1 
policy to protect human subjects involved in the planned 1 
activity. According to a Department official, the policy f 
-was issued under the general author ity/of the Public Health 
Service Act and was not a statutory^ requirement. 

■ y. ' ' 

MACOS was published before, the Foijri'dation voluntarily 
adopted the Department's, policy in October 1973. Officials . , . 
of ' the Found-ation' s 'precollege educational directorate 
and Office of XSeneral Counsel indicated that-/they we^e uncertain 
about th'e policy's intended application to. the E'oundation ' s a 
science education activities.' The Foundation's Deputy General 
Counsel 'subsequently reviewed "the Foundation' s statement 
on humam subjects as published in its grants administration 
manual and determined that it was not intended to be applied . 
to its educational activities. Her interpretation follows: 

. "Paragraph 272 of the NSF [National Science Founda- 
tion]. Gr,ant Administration Manual (NSF 73-26, Oct. 
1973) states that the safeguarding of the rights 
and welfare of human subjects involved in -activities 
supported by NSF gr"ants is the responsibility of the 
grantee institution. The paragraph goes on to state 
in effect that pending .publication of NSF guidelines, 
grantees are expected to follow the DHEW [Department 
of Health, Education, and Welfare] poricies on the 
protection of human subjects,, i.e. the regulations ' 
published by DHEW in May 1974. Paragraph 272 is so 
worded, however, as not to actually require NSF 
grantees to follow the DHEW- publications but rather 
to only suggest that they consider doing so." 



***** 



••* * furthermore, a close reading of the May 
1974 fej.nal DHEW regulations oh human subjects in 
- general the proposed November 1973 DHEW regu- 
•• *' rations specifically cove_^ring children, indicate 
quite clearly that the intent of the regulatio>is 

\ 
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is tb cove^r medical, behavioral and similar types 
of laboratory research (examples given in the> t 
guidelines include' drug research and research on 
diseases peculuar^ to" children) . Consequently, 
grants such as these made by the NSF Education. 
Directorate for the development of inst?:uctional 
materials were not intended to be covered by 
Paragraph 272 of NSF Grant Administration Manual, 
and grantees und§r such * * * grants are not ex- 
pected'^to fol-fow the DHEW reg.ulations f 
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The Fpundatign guidelines as proposed in paragraph 272 of 
its grants adminisFfation manual were not issued. Also, para- 

jgiLraph. 272 refers grantees-to the Department's December i971 
policy for guidance^ in applying the 'policy and not the De- 
partment's May 1974 regulations as cited by the Foundation's 

..Deputy General Counsel. 

The Department's policy became a regulation (45 C.F,.R. 46) 
in. May 1974. The regulations were issued under 5 U.S.C. 301, 
which provides general authority tOo the head of an executive 
department to issue regulations for governing his department, 
and. were 'applicable to all Department grants and contracts 
supporting research, development, and related activities in 
which human- subjects were involved, except those awarded by 
its office of Education and National Institute of Education 
unless specifically adopted by them. ..The regulations were 
not adQpted by these constituent agencies; however, their 
officials advised us that a policy for protecting human 
subjects had been and was being applied to their educational 
activities. We did not make further inquiries at these 
agencies to deterinine the /^extent of use of the policy orbits 
effectiveness in protecting human subjects. 



CONCLUSIONS 



* EDC conducted a comprehensive evaluation of MACOS. 
However, the significance of pre- to post-test gains attri- 
butable to MACOS was questionable because of limitations. in 
EDC's evaluation design. Furthermore, EDC's reporting of 
test results was incomplete because it did not adequately 
report factors which could have significantly affected 
evaluation outcomes. 

* EDC's evalua.tions did not compare achievement of MACOS 
and non-MACOS students ; however , planned evaluations will 
provide sifch data. This data would be useful to educational 
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users in selecting educational products and wo.uld perhaps 
have -greater impact if available when the product is Qri- 
ginally marketed. ' ; 

The Department's regulations for protecting^ human sub- 
jects ate aoplicable only to 4ts activities— adoption by 
other- Government entities is voluntary. The Department s 
educational units— the Office of Education and the National 
•Institute of Educatioa— are exempt from the ^regulations but 
apDly a palicy for protecting human subjects to .their, ed- 
ucational activities. Although the Foundation has adopted . 
the Department's policy) it does not apply the policy to its 
educational activities and has no formal procedures for 
safeguarding hum^n subjects involved in those activities. 

- RECOMMENDATION 

. we recommend that 'the Director of the Foundation review 
the need for establishing .procedCfTes to safeguard human sub- 
jects ihvol'ved ili the Foundation's educational activities. 

We are considering the effectiveness of education product 
evaluations in a separate review of nine other Foundation- 
sponsored science education projects. That review will provide 
'a more appropriate basis for any suggestions for • improving 
the Foundation'Teducational product evaluation-. 

■ 

AGENCY COMMENTS ^ . " ' ■ 

~- ^ ~ ' 

'fhe 'Foundation agreed with 'our recommendation and said ^ 
it would review the need for procedures to protect human 
subjects under its educational programs. The Foundation - 
comiUnted that its nonapplication of the Department of Health, 
Education, and Welfare's poli'cy fdi^ protecting human si^bDects 
■to. Foundation educational activities was in accord with De- 
partment thinking. The Foundation commented, however, that 
there are instances where the Office of Education and the 
National' Institute of Education apply the policy (not the 
regulations) to. their educational activities but not usually 
to their -curriculum development grants. 

"• As previously stated, we- did not determine the extent 
that the Office of .Education and National Institute of Ed- 
ucation w*ere using the policy for protecting human subjects; 
however, their officials advised us that they consider the 
protection of human. subjects in^ reviewing all grants and 
contracts .fpr educational activities. • . 
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The Foundation stated it was withholding^ comment, on I 
the effectiveness of educational evaluation because we are *^ 1 
consider ing >evaluation of Foundation-supported educationait ^ 
materials another study. The Foundation offered the . 
following- brief comments on our review of the MACOS evalu- 
ation, which it ha^3^ previously -discussed informally wi^th 

The Foundation commented that it differed with us as,/ 
'to the significance of formative evaluation ( testing per- . - 
formed on educational materials during development- to^^ 
determine needed revisions) as Opposed t6 summJitive eVal- * 
uation (testing o'f final educational mater ials to^ deter- 
.mine if objectives have been met). - Foundation of ficials^ 
believe the MACOS evaluation should not hav^ been expected ^ 
to account for the critical issues, we raised, si^ch as - ^ 
«stuc3ent maturation, because the evaluation was formative,. ^ 




suppoi 

gard^Less of th^ typfe pf .evaluation, we believe claims 
should not be 'made^ that are not supported -by the evaluation. , 

. The Foundation noted that our conclusions on/the MACOS 
evaluation were, reached aftec analyzing 1 of 78 findings in >^ 
EDC's evaluation report.)- The majority o'f . test results for 
MAGOS reported "by EDC were based' upon subj ective , testing - 
{"inter.vi(5ws with students^and teacher s*,' and 'observations of ^ 
^classrooip activities^). We re'^cognize the importance of sub- 
jective testing, as a mechanism in obtaining students' an^d 
teachers' opinions of materials beijrig developed. It produces 
results, however, which are based on persona! interpretations 
of responses from participants beio^ tested.. , ^ 

\ In a^proposal submitted- to the, foundation by the 
ey^aluators ptesently performing the- post-evaluation o-f MACOS, ' 
*the, following comm.ents were m*de about the subjective test- • 
ing performed during MACOS development: ' . ' ' > 

w* *' * There was heavy dependence made on tlje 
comments :of children and teachers' obtained through 
interviews to ^ signal the effects of the curriculum 
^ on, children's intellectual and psycho-social de-. 
veiopment. 'Protocols were interpreted from the - 
« frame of reference of the theories and beliefs under- 
"lying the curriculum. The protocols or transcripts' 
provided from, which interpretations were made &nd'* - 
inferences drawn^ are in some instances subject to 
quite other interpretations and inferences than 
those of * * * jtne* evaluators] . *. ^. *in some 
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cases, at least, the significance of the utterances 
"of students is in the eye of the beholder, not 
necessarily in the mental state or development • of 
the student. This is an inherent limitation of 
hoiv-experimental methodology. 

"The ways in which data are analyzed a«d interpreted, 
in some cases, or in which the basis, of their selec- 
tion for analysis or presentation was made, leave 
oAe unable ti> assess the implications of the data 

■in any objectively definable sense. * * *_^ne gets • 
a sense of intensive ex post facto hunting for ex- 
amples- that ara^coHsistenf witn the theoretical, 
ideological, and desigi^ belief s'&and<^h9pes of the 

- author's. - ^ | . 

* ' ■ - ■ ■ ' 

"* * * There is an equally st-rohg impression 
that the e^c p6st facto ' interpretations were 
derived directly, trom initial theory and- con- . 
ceptualizations-, although it is .difficult to 
demonstrate such a connection with confidence. 

"While innumerable instances ^ are given by * *-* 
*rthe evaluators] of. failures oj shortcomings of 
the course * * *, it is dif-ficult to discern a 
consistent scheme for deciding whether, the 
course is doing better or worse. It appears 
that an answer " inevitably would be ' . . .it 
all depends. 



0 
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A3 a result of the inhe^rent difficulties in, trying to 
interpret results from' subjective testing, we analyzed the 
claims made by EDC from the objective test ing^ per formed . 
which were -more widely distributed to the i,ntended users ' 
of MACOS through* a publishing brpchure* However, we. found 
that, for these, claims, the significance of pre- to post-test 
gains attr ibutable . to MACOS was questionable because of 
limitations in EDC'i^evalUation -design. That is, EDC did 
not account for, some factors th'at could ..have a significant 
effect on the test results, such as maturing of the student, 
the "testing" effect resulting from taking the same test -a 
second time^ and differences among teachers. 

Although we" only reviewed the claims made by EDC as a 
result of the pre- and post-testing "T!T&t were wi,dely diS7 
tributed, it would appear that all claims- made by EDC as a 
resul't -pf' the pre- to post-testing are questionable because 
of the testing factors EDC did not accoun.t^ f or . For ex- , 
ample', one .-claim make py EDC in its -evaluation Report was 
that on overall vocabulary competence, a 3D-pe.rcent increase. 



'19 



00027 



: . 'r 



1* ■>« 

from an- averag.3 40 percent to an averag'e 70 percent level 
of knowledge, ^'as found.. However, what percent of the 
increase was attr ibutabie'' to maturing of ti ^ students 
(such as from television, environment, and ,■ ther classes 
in'* school) and from taking the same test a second time 
cannot be determined since EDC did not account for thes^ 
factors. 

The rcundati'on commented that it and the MACOS evaluators 
■ "do not agree with our interpretation of one objective test ' 
claim made in the evaluation report and publisher's brochure. 
The specific finding they were referring to was in relation- 
ship to the EDC claim on the Man and Other Animals unit that: 

"Those students with poor aca<^emic background, found 
'SO often in the center city, gained in learning and 
mastery over the ideas and. concepts as much as those 
whose beginning positions were much stronger.-"' 

The MACOS evaluators, during our initial discussions 
with them, referred us to a table in EDC's evaluation report 
to supoort this claim. Our * evaluation of this claim is 
presented on pages 13 and 14. However, during our recent 
discussions with the MACOS evaluators and Foundation of- 
ficials, they said th.at we oversimplified the claim by 
placing too. much emphasis on the center city (inner city) 
portion of ' the claim. . " 

The MACOS evaluators statecJ that,, although the claim 
reads "found so often in the center city," they never meant 
- for the claim to be interpreted as a compari^son between 

center city students and students from'^other school systems. 
They stated tha,t the comparison was mcarit to be between 
students with poor academic backgrounds and students whose 
beginning positions 'were much stronger. However , regardless 
of what was intended, the claim as was actually pE.esented to 
potential -users could, we believe, cause a reader to assume 
that a cen^:er city "comparison was niadc and that those students 
would gain as. much , as students . whose beginning positions 
, : were much stronger. 
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• CHAPTgR 4 

■—■ ' T 

OBTAINING A COMMERCIAL PUBLISHER FOR MACOS 

■ The Foundation.' s policy for distributing -educational 
products stresses the need for comi)etition in publisher selec- 
tion. In accordance with this policy, EDC initially of f-ered 
MACOS to all interested publishers. After" some conditions 
had changed that could have influenced publisher interest/ ^ 
the Foundation did not require EDC to redeteri^me publishers, 
interest. This may have reduced the extent of competition 
in selecting a publisher. Also, EDC's reasons for selecting 
CDA as the MACOS publisher ,* as submitted to the Foundation 
for approval, ^were not totally 'supported. . - 

The Foundation, chose- not to review an EDC -contr-act with 
the MACOS publisher, CDA, providing services esseptial for 
marketing MAC0S;\thu5, the Foundation lost control of-f"|PS 
accruing under the ^co^tract from sales of MACOS." EDC^ anfCDA 
used funds from this, contract to support another EDC project 
by* obtaining matching funds from the National Endowment for 
the Humanities. The Endowment arguably did net have the 
statutory authority to match these monej^.s.. .. 

' GENERAL PROCESS -FOR OBTAINING 
A COMMERCIAL PUBLISHkk 

The Foundation's February 14., 1969, document "Policies 
for the Distribution of Publications and Other Materials 
Developed Under, the Science Education Programs of the Na^ 
tional science Foundation," which provides guidance for the 
distribution of education materials developed with its funds, 
included the following: 

"Policy must * * * achieve maximum educational 
benefits for ail potential users. Educational 
benefits will take precedence over all other con- 
sideratio.ns, including possible generation of 
income. > • ^ 

"* * * public and private interests must be safe- 
guarded. [An example of public interest is the 
■ continuing availability of. the education mate- 
rials ] * * * In the private sector, the concern 
' is avoidance, insofar as practical, of interference 
with normal commercial practices * *■ * . For in-.. 
stance, there should be open competition among 
* * * organizations for distributicr. rights to 
./ project materials. Agreements witH- distributors 
should provide for price levels consistent with 
the existing competitive price structure. Zn the 
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case of commercial organizations royalties ai?d 
other conditions should be adjusted to allow a 
reasonable but not undue profit."'. 



* * The grantee, as owner of the materials, 
may ^negotiate and enter into contracts with dis- 
tributQrs. The steps leading to negotiation as 
well as the negotiations are morfitored^ by NSF 
[National Science Foundation], and resulting. con- 
tracts are subject to NSP approval, * * *!* 



"The fyxancial interest^ of NSF * * * in all mate- 
rials .jshall be in proportion to financial support 
. by NSF; of the materials or projects 

When the'' grantee and the Foundation agree that the edu- 
cational mat;erials developed are to be printed and published 
commercially, the grantee develops a plan for obtaining a 
publisher that the Foundation inust apprpve. The plan gen-, 
er^ally consists of the grantee's hotif ication, usually in 
trade journals and through a mailing list, to prospective 
publishers that thQ materials are available for publishing.* 
The grantee tfien holds a conference with interested pub- 
lishers and solicits formal requests "^to publish the materials. 

The grantee is responsible for evaluating the proposals 
received and submitting^ the selection and reasons for it to 
the FpundatiQO for approval. The grantees are provided with 
the Foundation's ddcument pn the broad policies for distribut- 
ing materials! ' However, the Foundation has no specific 
written criteria^, such as defining expected publisher capa- 
bility and reasonable f)rofit, for grantees to consider in se- 
lecting a pubipsher. When the Foundation's approval is ob- 
tained, the gr'antee negotiates a contract with the publisher 
which is subject to review and- approval by the Foundation, 
although the Foundation does not become a formal party to 
the contract • , • . 

AWARD OF PUBLISHINb CONTRACT 



Beginning in 1967, EDC tried to obtain a commercial 
publisher for 4AC0S^ and complied with the Foundation's re- 
quirements for selecting a publisher by advising publishers 
that MAgoS was available for publication, through such means 
as advertisements in trade journals, and by setting up a bid- 
ders' conference. 1 . - ' 
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During JL^67^ EDC contacted over 50 publishers and met 
separately wi^h 43 that indicated some interest ia MACOS.; 
however, no publ-^^her was Villing to market ^COS. ^The 
publisher 'reaction to MACOS was unfavourable apparently be- 
cause the ^ <f ^ 

--special concepts were unconventional , 

— special .trainiftg was required for teachers, 

--project materials were injriovatiye, afiu 

— films and other matejrials were costly.* 

Because no cofflm:erciai publishers willing to market 
MAC03 could be found, in'june 0.969 the Foundation authorized 
EDC to commercially publish anfl distribute the materials 
in an effort" to demonstrate t^ie commercial feasibiiity and 
profitability of marketing MACOS. EDC had: sales of^ about 
\1$578,000 at cost during a 17-uionth period. 

In the spring of 1969/ the FouH.dation decided to accept 
a Jow royalty rate to help obtain 'cottimercial publisher for 
MACOS and" to keep from adding further cost- t6 the expensive • 
MACDS materials. The Foundation ^royalty rat*s,were: (1) 
3 percent. for "text materials, (2) 5 percent (for .four Man and 
Other Animals Super' 8 f ilms^ and (3) 2-1/2, pVcent for six 
Net'silik EsKiroo Super 8 films.. 1/ Officials of the Founda- 
tion 'and EDC acknowledged that the Foundation's rates were 
low. 

Although the^ov; royalty rate was designed to att^'^s^ 
a commercial publisher and some trial sales results were 
available to demonstrate a demand for MACOS, the Foundation 
did not require-^EDC to formally advertise this change or to 
convene another publishers' conference. Broad dissemination 
of the royalty rates" and the sales results might have in- 
creased, the competition for a MACOS publisher. 

Eventually five commercial publishers showed serious 
interest in publishing MACOS; however, one—Initial 
Teaching Alphabet — did not submit a .proposal. The other 
four submitted proposals on the following dates: . . ^ 
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l/In addition, the National Film Board of Canada, a joint 
~ producer of the.Nfetsilik films, received a 5-percent 
royalty for these films, .and five commercial film rights 
holders received royalty rates ranging, from nbthing-to 
30 percent, or an average rate of about 19 percent. 
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International Learning Corp. , . - • • ■> 

Fort Lauderdale, Fla. June 25, 1369 

KDI InstrtictiqngJ. Syitem, Inc.,\^ 

Columbas, Ohio., • 2r^6ecember 1969 

Westinghouse^ Learning Gorp.,. — 

New York, N.Y. " January 30, 1970 

.* . - * 

CDA, Inc., ■ . ^ - 'oo iQTn 

Washington, D.C. ' - February 23, 1970, 

According to the MACOS' project director , the five pub- 
lishers were provided with the predeternvined royalty rate, 
''so it was not a factor in the negotiations. ^ <• " 

Morvitorinq publisher selection ^ . ' ' 

■International ^Learning and KDI were initially eliminated 
from the competition, although EDC's records do not document 
the reasons for the decisions. Also, EDC's basis for select- 
. ing CDA over Westinghouse Learning in March 1970, as, submitted 
to the Foundation for approval, was not totally supported. , 
EDC's four selection criteria provided the Foundation and our 
evaluation thereof follow. 

— The first criteria was the ability to perform the '\ 
mechanics of publication. EDC advised the Foundation 
that both finalists had this ability, but an EDC of- 
ficial told us that Westinghouse Learning had more 
publishing experience. - \ 

— The second criteria was^ financial ^Eapability. EDC 
did not determine the financial capability of either - ■ 
finalist before selecting CDA. . . 

The other two criteria concerned dissemination plans 

and teacher training. EDC informed the Foundation 
that, although each finalist had these capabilities," 
CDA had- (1) shown greater, interest in considering 
alternative dissemination designs and (2) bet-ter 
staff capability to implement teacher training re- 
quirements. However, EDC could not provide documen- 
tation to support these conclusions. 

Accordingf to an EDC official, the decision to select CDA 
was ultimately based on EDC's confidence in the people of 
that organization. However, this was not. one of the specific 
criteria EDC communicated to the Foundation to justify .select- 
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Moreover, we were advised that a part-time EDC employee, 
instrumental in developing EDC's MACOS dissemination plans, 
also assisted CDA in formulating its MAGOS dissemination 
strategy. Further, the responsible EDC official advised us 
that this individual's fav6rable assessment of CDA f capa- 
bilities was one of the factors considered m selecting cda 
as publisher. Such a situation may have given' CDA an 'advan- 
taqe over other interested publishers and should have been 
disclosed to the foundation. Details of this situation are, 
included in appendix I. 

PROFESSIONAL SERVICES . CONTRACT ■ ' ' 

BETVfEEN EDC AND CDA - ' 

■ In seeking a -publisher , EDC considered' the requirement 
for teacher training in the use of the. MACOS materials as a 
necessary part of any publishing agreement. Concurrent witn 
the signing of the MACOS publication contract on July ^u, 
1970, EDC and CDA entered into a separate contract, referred 
to as a professional services contract, in which they agreed 
that tbie following services would be provided: • 

"1 During the term of bDA's license to publish 
said materials, EDC agrees to use its best efforts 
' to assist CDA in its dissemination efforts includ- 
ing the utilization of its staff and" resources m 
activities of the following type: , 

a. , liaison, with school? and school systems, col- 
/ 'lege and university pre-service training pro- 
grams and witJh appropriate private ■ and , govern- 
mental agencies concerned with curriculum and 
sLaff development . - 

b. identification of present or prospective users - 
6t the Work- and leadership "teachers trained m 
summer institutes 

c. assistance in the development of^eacher- 
training programs (both in-servile^ and pre- 
service) ' . I - ■ , 

d. joint development of a communications and in- 
formation exchange among educational institu- 
tions using the Work. 

e. revision, modification or supplementation of 

, the Work , ' ^ u 

f. continuing research and ev,aluation of the ex- 
tent to which the Work is accomplishing ' its 
objectives , 

g. these and other efforts to be undertaken shall 
be determined by EDC in consultation with ' 

/ CDA. 
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2. It is understood and 'agreed that the above 
^will be undertaken solely within the limil: of 
• funds made available by CDA to^EDC as herein^ 
after set forth." 

To finance these services, the professional services 
coritjcapt provided for CDA to pay EDC half of its promotion- 
dissemination' budget.. The budget amount. was to at least 
equal' that of the normal , commercial practice for dissemina- 
ting multimedia educational materials. , According to EDC and* 
CDA officials, the service reimbursement rate was actually 
esfeablislied at 15 percent of gross receipts from MACOS ma- 
terials salesr ^ ■ • 

Before finalizing the/contract , EDC advised the Founda-. 
tion of the proposed cdnjiract, the services to -be provided 
therein (except for provision "g," which was added later), 
and the general reimbursement provisions (not ttie 15-percent 
rate). The Foundation apparently agreed' in principle to t^e 
arrangement but chose not to foirtnally review the contracts 

Foundation grant adm^inistration requirements in effect 
'when MACOS was being developed and published, and applicable 
to EDC under its grant, provided that the Foundation must . 
approve any contractor subcontract, including any , amendment 
thereto, , before it was si'gned. In this respect, the Founda- 
tion's July 1969 publication, "Grants for Ed^ucation in 
Science," stated:, 



"* * * Such approval will be based on a determina- ' 
tion that the contract~or~FQbcohtract contains the 
provicions required to protect the grantee's and 
the Government's interest, including the right of 
audit of expenditures and income associated with 
the contract. ****** * 

Current Foundation gnant administration procedures also con- 
tain .these requirements. 

The Foundation should have reviewed and approved the 
professional services contract to insure adequate protection 
of the Government's interest. In addition, (1) 'the teacher 
training services to be provided were viewed by EDC as es- 
sential to marketing^ MACOS and (2)' the contract was a some-, 
what innovative arrangement in which the Foundation had 
only limited exp)erience.N 

Implementation of the contract • ^ . 

During July 1970 through December 1974, CDA's cash 
receipts from the sale of MACOS project films and materials 
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V'^^^ totaled about ?2.6 million, of which about $387 ,000 accrued 
to EDC 'Under the professional services contract. As of 
May 6, 197-5, EDC had spent about $31^,000 , of which a^out 
$119,000 was used for activities- related to the MACDS- 
project, .such as: ' 

— Evaluating CDA's teacher education and dissemination 
program for MACOS. 

— Developing procedures and materials- to be used in 
MACOS teacher training workshops. 

1 — Developing a film on the modern Netsilik Eskimos. 

.The remaining funds EDC expended— $195 , 000~were used 
■ for the following purposes not formally related to MACOS, 

People and technology .project 

In June 1970, the National Endowment for the Humani- 
ties 1/- granted EDC $600,000 to develop "People and Tech- 
nology,." a social studies project to produce educational 
materials for grades five through seven. At that time 
the Endowment advised EDC that it would provide ad,^itionai 
matching funds for the project if donations c,6.uld 'be ob- 
,tained by EDC from nongovernmental sources. ^/ 

In this respect 20 U.-S.C. 959 et seg . provides- for the 
Endowment to accept and match restricted gifts. However, 
the Endowment's publication "Gifts and Matching Background 
Information for Institutions and Organizations" .provides:^ 

'"The Endowment will not match a restricted gift 
from the institution conducting the project for 
\ 'which the gift is intended, nor from persons or 

other institutions involved in the project; the 
Endowment will not match a restricted gift from 
Federal funds nor from current or pending recip- 
ients of Endowment grants." 

The Endowment's General Counsel advised us that the limita- 
tion on .receiving restricted gifts from the institution con- 
ducting 'a project or others involved in the pro.ject was 
- adopted because of the -belief that such moneys would not con- 
. stitute a gift in law. 



1/Part of the National Foundation on the Arts and the Hutnan- 
~ ities, a Federal agency created to promote activities to 
support and -disseminate knowledge in these disciplines. 
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In September 1972 EDC and CDA determined that EDC's 
assistance in marketing MACOS was no longer needed and that 
funds accrued under the .prof essiori'al services contract could . 
be used for other purposes. , They informally agreed that CDA 
would pledge funds so accrued to the'-Endowment for matching 
purposes for the People and Technology project. 

Between-October 1972 and. December 1974, CDA pledged 
$205,000 of the -professional services cpntract funds to the 
Endowment as a restricted gift for the People and Technology 
project-. As of June 30, 1975, CDA had made payments of 
$17 570(>Q which were matched by the Endowment, resulting in 
a retur-nN>ifi|350,-000 to EDC, and a pledge. of $30,000 was out- 
standing. " . o 

• We question* .whether the sums remitted by CDA to the 
Endowment 'under the informafl amendments to the professional 
services contract with EDC were within the Endowment's match- 
ing authority* , CDA claims that it waived EDC's obligation 
to perform^ under the services fcontr.act . to the extent that 
moneys^ therefrom were, paid to the Endowment. According to 
.CDA> EDC did not waive the right to receive those moneys and 
CDA was always obligated to. pay them to EDC or to the Endow- 
ment for the People and Technology project. EDC maintains 
that these moneys were obligated only if both parties agreed 
oh a dissemination plan each' year. The contract, however, 
does not condition CDA's payment. obligation upon such an. 
agreement and contains an arbitration clause to.deal with any 
disputes., ' * 

Since the "qifts" were made by CDA discharging its obr 
ligation to EDC under the services contract and in ef^fect the 
moneys were simply returned to the party to which they would 
otherwise have been payable, no bona fide gift occurred. 
These moneys were thereiEore arguably outside of those cate- 
gories of funds permitted' to be received for matching by the . 
Endowment pursuant to 20 U.S.C. 959. Alternatively, since 
EDC had the legal right to these moneys, it could be suggested 
that EDC actually made^the "gifts" through CDA. However, re- 
stricted grfts from grantees conducting the project for which 
the gift is intended cannot be accepted for matching under 
the provisions of the Endowment 's publication (previously 
cited on page 27). The transaction^ >was thus arguably improper 
under this view as well. \ 

\ 

EDC maintains that the funds were matched^ by the Endow- 
ment with full kno.wledge of their source o^n the basis of an 
October ^6, 1972, letter from CDA to the Endowment which of- 
fered the* pledge of funds for the People and\ Technology 
project. The letter stated: 
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* * bur understanding and agreement P^oy^^^d 
that in addition to normal royalties on the film 
and print materials * * *, we would provide EDC 

fts use- ah .unspecified sum f ."'oney each 
year for continuing review and development of the 

programiparticularly in the/reas of teacher 

education and evaluation). ' > 

I 

"* * * we' ace pledging to the National Endowment 
For The Humanities a contribution in the amount 
■of $100,000;for support of fe. .'People and Tech- 
nology' unit now under development at EDC. Tnis 
Sfan? is ini lieu of a like amount which would be 
Lde under ithe previously -mentione^ SlN^oroS^am 
EDC services in connection with the MAN prpgram. 



* * *•• 



ThP Endowment's General Counsel said that, in consider- 
ing CDA's'Scrblc'e! 1972, o"er,.;he interpreted it to mean 

S ^nrow^^n^ did/not review t^^^^^^^^^^ ^^^^ 

llll iirZlll nft ceil;: lutu^rspecial consideration from 
ED? when People and Technology was offered for publication. 

" Diversity in «-hp firhool Community" 

iHe Off ice of Education's program; it was not formally part 
of the MACOS project. 

RELATIONSHIPS BETWEEN 

THE FOUNDATION, KDC. AND CDA 

we were requested to review the relationships between 
r, A FTiC and CDA. We determined the three 



EDC's. annual reports ^ ^inutes of board meetings/ and 
personnel files" and CDA's lisVings of officers and- stock- 
holders and (2) compared a listing of EDC and CDA officials 
to /the Foundation's employment termination records dating 
back, to about 1970 and its* employmejit records for personnel • 
on board on May 31, 1975. We found no indications of so- 
called ''interlocking di,rectorates"* between the Foundation,- 
EDC, and CDA &r other questionable personnel relationships 
except for that discussed on pagle 25. Details of this review 
are^ included in appendix I. 

CONCLUSIONS^ , ■ • 

The Foundation shoul<3 monitor the selection ^of pub- 
lishers more closely to insure that all competitors a^re 
informed of vital in forma tlOTi, such las predetermined * 
royalty urates, that could in^uenqe (1) a competitor's 
decision to" submit a proposal and (2) proposal contents. 
Closer monitoring^ should include a detailed review of pro- 
poaa^is to insure that reasons for publisher selection are 
valid and d'ocun\ented. . * 

The Foundation should be particularly vigilant in moni- 
toring publishing arrangements where low royalty rates, and 
oth^r non-routine arrangements, such as the prof ession^*i^**serv- 
ices contract,* are 'permitted, to insure adequate protection 
of the^Government ' s interest. In this respect, the Foundation 
should have reviewed the professional services contract since 
It provided for teacher training services considered essential 
to marketing MACOS. Had the Foundation reviewed and approved 
the professional services contract,' it could have- provided for 
the dispcciti^n of the accrued income after services under the 
contract were deemed no longer necessary. 

- The ^175-, 000 remitted by CDA to the Endowment under the 
informal amendments to the services contract arguably was 
not donated as a matter of law a^nd does not constitute a 
gift., - ' ^ 

RECOMMENDATIONS 

We. recommend that the Director of the Foundation 
strengthen pbotf^dures for selecting publishers of educa- 
tional- mater ial^s to insure that: 

— ^^ublisher interest in marketing -educational- materials * 
is redetermined when conditi^ons Which could affect that 
interest change*. - * 
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--Documentation to support the reasons for publisher ^ 
selection is required. ' , 

* 

—All contracts and subcontracts for 'tnarketing educa- 
tional materials are reviewed and approved. 

* ^ * 

we recommend that the Chairman, National Endowment for • 
. the Humanities., review the Endowment's transactions With EDC 
and CDA to determine (l^ whether the f^^^^f ^iven b/ CDA foe 
the Peoplfe a,Qd: Technology project represented a legal gift 
within the authority of the Endowment to match 1/ »a^d. (2) if 
net, whether the matching funds the Endowment gave to EDC 
should be recovered. , . . - ' 

AGE NCY CDMMENTS ' • ' - 

The Foundation agreed with ou'r recommendations and stated 
- that changes needed to implement them are being developed. 
^ in addition, it provided the following comments. 

' • , ^ * 

I / According to the Foundation, our report' imp;Lies ^the.need 
< for /its direct involvement in the operations of its grantees 
and/contractors. The Foundatj.on cited as an example the_ 
statement i.n chapter 4- that it has no spec#f ic^ criteria for 
grantees to consider in' selecting a publisher. The Pounda- 
?i4n recognised the need to improve administration of .ts 
ciLiculum development programs, but Relieved it should b. 
'primarily involved in establishing policy to guide grantiees 
aria contractors to carry out programs. ^ 
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.J we agree that providing policy guidance to grantees and 
cAhtractors is a primary Foundation role; however, that -^fo^e , 
nLds to be supplemented with adequate management control-^ 
Jf insS^e that'^grantees and 'contractors are f oll^v.^^^ 
Ssiness- practices. For' example', our re.view showed that the^ 
K^Sns eEc gavfthe. Foundation for selection of CDA to pub- 
Hsh S^COS were not totally . supported . • Further,, the selection 
'of a publisher for a Foundation-supported curriculum is a 
•major project milestone in which both th6 Toundation and the . 
.developer should consider specific needs to disseminate the 
curriculum before the developer selects a publisher - and the 
Foundation approves it. • . ■ ► . 

The Foundation also commented that the question of 
whether it should have reviewed and approved the' 'prof essionai 



l/A Julv 14, '1975, Endowment tequest for an opihion from the 
->?ompt^oll4r General under .31 U.S.C". 74 as to its authority ; 

to ma^ch CDA's outstanding pledge of ?30,000 for People . 

and Technology (see page 28) is pending.^ ■ ' 
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services contract between EDC and CDA under the ^cope of its 
July 1969 publication •^Grants for Education in Science** is 
a matter of interpretatioa, which the Foundation believed it 
handled correctly. ' v 

The July 1969 publication stated under the caption 
••Contracts and Subcontracts Under a Grant" that; • 

* "If it is' necessary .to enter into contracts or sub- 
contracts, particularly for commercial production 
of materials, the Foundation must be apprised in 
advance of the cdnside'rations and procedures^, which 
- are , planned for the selection of the contractor. 
All appropriate and interested concerns should 
have an opportunity to -submit a proposal, and 
s^lfec.tion should be based on such considerations, 
as suijl^ability of contract conditions, quality, 
distribution ^and price. The Fourtdatiop must ap - 
prove, any coritract'pr subcontract before ^it is^ 
^ signed . Such approval will be based on a :deter- 
V minatxon that the contract or subcontract contain3 
the provisions required to pJrotect^the grantee's 
and the. Government 's interest, inclXid^ng the 
right of audit of expenditures and income assp- 
ciated with the contract. \ The Foundation must 
also approve any amendment to. any such contract 
or subcontract including thoseerto be made after 
% * ' termination of the grant." (Underscoring, added.) 

The larighuage of the July 1969 publication appears to 
clearly require that l^e Foundation approve any contracts" 
or, subcontracts under a grant. Nevertheless r>egardless 
of interpretation, . sound -business practice would, seeifiihgly 
have dictated that ^ the ^ofessional services contract; — 
which provided services considered vital by the MAGOS de- 
veloper for successful use of MACOS — should have /been re- 
viewed and approved by' the Foundation. Sc^h arguments, . 
however , j^appear academic because the Foundation has agreed 
with our recommendation that it review and approve^ all con- 
tracts and subcontracts for marketing educational materials. 
' » » 

Th^ Foundation commented that in chapter 4 w6 stated 
that' the low royalty rate used for MACOS .printed material 
might have improved the competition for a publisher. The 
Foundation, however, does not believe that the low royalty, 
ra'e would have* influenced publishers that once showed lit- 
tle interest in publishing MAGOS to irecohsider. 

To^ comment on the ef.fect of the low royalty rate in the 
publisher selection process for MAGOS is r&ther speculative, 
alb^iough the two finalists for the MXgoS publishing^ contract- 
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pr.edetermin.e.d. In addition, EDC^nd a low roy- 

Idvised us that the Foundation decided to accept a low roy 

al-ty rate to help obtain a commercial publisher and to not 
add further cost to MACOS. t> . , 
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CHAPTER' 5 



IMPtiEMENTATION OF MACOS 

The Foundation •s-precollege instructional improvement 
implementatioa section makes awards for* the iroplementation 
of major curriculum and course developments at the precollege 
level to strengthen school science and mathematics programs. 
However/ the Foundation lacks definitive criteria concerning 
how long and to 'what extent precollege curriculums will be 
funcied"; Projectis that have obtained ,a commercial publisher 
are eligible to compete for and have received Foundation 
implementation funding. 

Proposals requesting implementation funds^ are , subjected' 
to a ^competitive review process in which the Foundation is 
assisted by ad hoc panelists in evaluating the proposals. 
Foundation officials place. much weight on their own profes- 
sional judgment in determining which proposals- to fund. 
They^ 'however ^ do not always document in the files their 
reasons for supporting or denying , funding for proposals. 

The Foundation, which requires /a statement on all pub-- 
lished materials Ideveloped with Foundation support acknowl- ♦ 
edging, its support- and stating that the views in the material 
are those of the author and not necessarily shared by the 
Foundation, has no routine procedure to insure that such a 
statement is in fact included. 

PROCEDURES FOR AWARDING 
PRECOLLEGE IMPLEMENTATION PROJECTS 

An annual competitive evaluation review is conducted to 
select proposals to receive funding fojc^ implementation proj- 
ects. Proposals submitted, whether naV^or renewals, must 
compete with other proposals submittecilsthat ye^ar^. B '*th 
Foundation-supported .and non-'Foundation-suppopted cuiriculums 
are eligible for implementation support. The Foundation does 
not have 'definitive criteria concerning' how many /years and 
to What extent curriculums "should be eligible for funding.s^"^ 
Therefore-, projects could conceivably be commercially mar'*- 
keted for a great number of years and st^ill be eligible to 
compete for the Foundation's implemebtation funding. ^Accord- 
ing tO'Foundation officials, implementation funding is pro- 
vided for projects that, have been commercially marketed f^r 
a number o£ years to provide for sucji^ things as teacher turn- 
over and additional school systems adopting the curriculums. 

Program announcements are sent annually tt> th'e educatibn 
and scientific communities conveying the Foundation's guide- 
lines for obtaining a precollege" implementation award. The 
guidelines are distributed to all school systems with 
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• enrollments of over 10,000, s°">^^^^f^^.^"!L Katio^a^ Science 
colleges and universities, individuals on ^he National Science 
Teachers' Association supervisors' mailing l^^t, directors of 
current projects, individuals, submitting proposals in the 
previous fiscal year , and individuals requesting the gui^^ 

lines. - According to Foundation o"^"^l = ' ^P^5°f,i^^^li^ 
20,000 implementation. announcements were ^^^tributed in 

fiscal yeSr 1975. In addition, other F°""^^t^°%P"^ii"Ji°"^ 
describe, the science education activities and reference the 
documents that interested persons may request. . 

Announcements in the Foundation's guide for preparing- 
proposaJs and operati^ng projects for instructional improvement 
implementation describe the precollege PJ°f 
and invite proposals for projects to implement ma]or curricu 
' lum ^nd course developments at the precollege level in natural 
a^i social sciences and mathematics. Theannouncemen^ guide 
aiso defines th'e ^eligible institutions, which are generally 
.?he sa^e as tho^e fo? the materials <and ins.truction develop- 
ment section -as stated on pa.ge 5. 

The implementation announcement guide also outlines bro^ 
cateqories in which the Foundation will consider awarding ^ 
projects F^r example, in the fiscal year 1975 preoollege 
instruction improvement implementation guide, some of the 
brbad categories included familiarization With alternative 
curriculums or approaches, training of re^ 



long-term dissemination and maintenance, and installation in- 
a Significant segment of a. school system. Also listed in the 
fiscal year 1975 implementation announcement guide are the 
target groups .which will- be affected by. the implementation 
activities. These groups include: 



—Leadership specialist projects, which are directed 
toward individuals who influence curriculum decisions,, 
such fls principals,, supervisors, college faculty, super- 
intentents, and curriculum directors. $h.e projects \ 
are designed to provide information to school decision- 
makers about new mater ials, or to develop expertise so 
that leaders can organize more massive, local implemen- 
"tation efforts. . / ' 

—Teacher projects, which are designed to bring about 
classroom change or improvement through effective 
teaclier use of new instructional materials or prac- ^ 
tices. ^ 

—School system projects, which are directed at bring- 
ing about specified curriculum or course changes in 
classrooms wherie schools and -solhobl systems are 
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willing to make conunitments of funds, personnel, and 
other resources* 



^ Proppsals -received are considered unsolicited and are 

evaluated in a competitive process for funding. A closing 
date is established for, .receipt of proposals. Pr<^posals are 
initially grouped by the education directorate's program 
objectives (such as careers in science or scientific liter- 
- \ acy). Within the, program objective groupings,, the proposals 
/are grouped by type of activity (leadership specialists, 
teacher^ or school^system projects ) ; grade levels (elementary 
and/or secondary^Ty and curriculum, where possible, (such as 
; MACOS). Fpf example, one grouping for proposals may be 
\ literacy-school systems prpjects-elementary-MACQS. 
> . * ♦ ^ ' ^— > 

In reviewing the proposals, the Foundation is assisted 
; by advisory panels of scientists and 'educators drawn fro..: 
colleges, universities, school . systems, professional socie- 
^ ties, and other nonpr.ofit or ganizations, or Federal agencies" 
\ concerned with science education. Foundation staff match 
^-'panelists' qualifications to proposal content. Ail pro- . 
'posals are read by two panels^ each/ consisting of three in- 
dividuals. The panelists rate the proposals on the fbllow- 
. ing sevon criteria: objectives (reasonable-unreasonable) , 
needMgj^e^it-little ) , plan (suitable-unsuitable), staff 
(strong-weak), commitment (adequate-inadequate), impact 
potential (substantial-insubstantial), evaluation design 
(significant-insignificant). The panelists rate each crite- 
rion on a scale of 1 to 7; one is unfavorable and seven 
higjhly favorable. As a result, the highest possible score 
an implementation proposal can receive is 294 (7 criteria x 7 
the highest possible rating for each criterion x 6 panelists). 
Average , panel scores range f£p'm 160 to 170... \^ 
^ ^ » *■ 

After' the panelists have studied several proposals and- 
tentatively evaluated them, the panel discusses each proposal. 
After the discussion, a second rating (which may or may not 
be the same as the initia^l one) is made by each panelist. 
The second rating gives the panelists an opportunity to dis- 
cuss the* proposals and to possibly obtain and consider ih- 
• formation that they may not have initially considered; 

The recommendations of the panels Are an important 
element in the Foundation's determination of which propo- 
sals to fund, but the final decision is made by the Foun- 
dation, after reviewing the proposals and the panel ratings 
and panelists' comments. The Foundation generally approves 
.grants to proposals rated highest by the panels and denies 
grants to those rated »' lowest . Disposition of proposals ' 
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.given intermediate ratings will be based on ^^^^ PJ"^J 
scores and comments, but also program balance with respect to 
disciplines, grade levels, curri^ulums, institutions, and geo- 
graphic distribution. 

Program managers' recommendations for grants and denials 
^re reviewed and approved or disapproved, :^n turn, by the sec- 
tion head of 1:he precoHege instructional improvement in>Plen^en- 
tation section, by the Division Director for Pre-College Educa- 
tion in Science, and by the Assistant Director for Science 
Ed^ation. Denied proposals are signed off at the division 
level and approvals require the signature of the Assistant 
Director for Science Education. ' ^ » 

IMPLEMENTATION FUNDING * . • 

Our limited review of the Foundation's process for awa'rd- 
inq precollege instructional improvement implementation grants 
showed that proposals are competitively- evaluated- separately oy 
ad hoc panels and Foundation officials. Acco^rding to Founda- 
t ion of f ic-irajrs— wh-i-le-the panelists aid and are the major in- 
gredient in the evaluation process, the Foundation makes the., 
final' determination. We reviewed the panels' ratings of the , 
implementation proposals the Foundation received for fiscal 
vear 1975, for which recommended awards totaled about ^-L^-o- 
million. Of the 621 proposals submitted, 306 were funded and 
315 were not. Of the 315 not funded, 289 were denied, 8 with- 
drawn, 5 deterred for later action,!/ and 13 not eligible for 
support.- <. - , ■ 

As previously noted, the ^highest panel score a proposal 
can "receive is 294- and the average panfel scfores are reportedly 
between 160 and 170. 

^' In analyzing the detailed breakdown of the panels' rat- 
ings, we sche-duled, i'n intervals of '25, the panel scores for 
the 621' proposals. In some 5cases,.for proposals receiving 
panels' ratings within the same range, many were funded while 
many others were not. For example, for the school systerri 
projects (projects compete within project categories ), 37 pro 
posals were rated by panels at between 151 and 17,5.,. yet ly 
were funded and 18 were not. Thus, Foundation officials 
place much, .emphasis on their professional judgments in fi- 
nally deciding which proposals to fund. The overall panels 
ratings are' presented in the following table. 



l/We did not determine the final disposition of these propo- 
"sals and for purposes of this report considered them not 
funded . 
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Intervals of panels* r atings / 
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Below to to to to to to to to 
Project categories 75 100125 150175200'225.250275 - Total 

■ (Number of proposals) 



Leadership specialist: 
Funded 

Not funded I 
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1 


5 


13 


29 


21 


. 2 


71 


1 


5 


7 


30 


20 


13 


5 


0 ' 


81 


1 ^ 


2 


1 


8 


35 


43 


15 


3 


a/198 


1 * 


9 


28 


56 


42 


10 


1 


0 


a/149 


0 


0 


1 


19 . 


45 


42 


18 


1 


126 


6 


10 


31 


18 


10 


5 


0 


2 


84 


1 


2 - 


3 


32 ^ 


93 


114 


54 


6 


a/305 


8 


24 


66 


104 


72 


28 


6 


2 


a/314 



Teacher-centered: 

Funded O- 
Not funded 2 

School systems 

Funded , 0 
Not funded 2 

Total: 

. Funded 0 
Not funded * 4 

a/Totais do not reflect one teacher-centered project reviewed for 2 years funding and 
one not funded because it was ineligible for support. Panel scoreswere not re- 
ported for 'these two proposals*. 



Both Foundation-supported and non-Foundation-suppor ted 
curriculums are eligible to receive Founda.tron implementation 
funding. For fiscal year 1975^ Foundation-supported curriculums 
received 84 percent of the total obligations. We attempted to 
compare the percentages o^ Foundation-supported and npn- - 
Foundation-supported curriculums receiving support to the per- 
centages of those groups requesting support/ but according to 
Foundation officials, this information was not readily avail- 
able . 

MACOS implementation funding 

..... 

During fiscal years 1967-75, the Foundation awarded over 
$2.3 million in MACOS implementation grants to various colleges, 
universities, and other nonprofit institutions, including EDC. 

Wev identified the MACOS implementation proposals the Founda- * 

tion received during fiscal year 1975 arid noted that, of the 37 

proposals received containing MACOS (either wholly or in part), 

19 (or 51 percent) were funded jind 18*(or 49 percent) werg not. 
For the implementation, proposals funded, the panel scor.es 

ranged from 175 to 237, and for those not funded*, the scores 
ranged from 106 to 248. 

We identified the fiscal year 1975 non-MACOS proposals that 
we're not funded but received higher panel scores than proposals 
cqntaining MACOS that were funded. Forty non-MACOS proposals 
were not- funded but received higher panel scores, ranging from 
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176- to 229, than MACOS proposals that were funded. J^^^^^^^P^^' 
during fiscal year 1975, for the category literacy-elementary 
school system projects., seven non-MACOS proposals that were not 
funded received higher panel scores., ranging, from 185 to 225 , 
than a MACOS proposal that was funded received a panel 

■score of 182. According to Foundation of f ^^^^^^^ ' . ^^^'^^'''"^ "^da 
also -occurs for non-MACOS curriculum proposals, since the Founda 
tion makes the final determination as to which proposals will 



be funded. 



we selected three of the proposals having the ^^^^^est 
variance between panel scores and the score of the MACOS-funded 
proposal to obtain the Foundation's justifications for nonsup- 
por?? The justifications given by the Foundation's program, man- 
agers for not funding the non-MACOS proposals were: 

—The institutipn had submitted six proposals and four 
wer e funded . 

■-The number of teachers to be trained would have been too 
great and nothing indicated that teachers wanted to use ; 
or schools would purchase the materials.. 

—The Foundation decided o.nly a certain sum of money 
would be spent on implementation funds for this spe- 
cific curriculum in the, school system proDect cate- 
gory and the ceiling had been reached with proposals 
considered to be more worthy of support. 

'For the last example, the Foundation had established a 
funding level of between ?150,000 to $200,000 for this cur- 
ricuium in the school system projects category. As a resuU, 
ti is proposal requesting $26,299, to implement the curriculum 
in a specific school system was not funded because other pro- 
posals lor this curricilum were determined by the Foundation 
?o be more worthy of support, were selected for funding, and 
already exceeded the funding ceiling by ^15,000. 

Foundation proposal files had documentation justifying 
the reasons for the denial of one of the proposals and a 
Foundation program manager gave us the reasons for the denial 
of the other two. Program managers are not required to file ^ 
written justifications explaining why proposals were or wece 
jiot funded. In some instances program managers may prepare a 
diary note for the file explaining why a proposal was not 
funded. But written justifications for denials are generally 
Dreoared only for those proposals in which the proposer re-, 
^ auested such a justification. For these requests, tne pro- 
aram manager prepares a standard form fof the files noting 
the reasons for denial. The requestor receives this .lafotma- 
tion by telephone. 
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EDC curriculums funding 



EDC curriculums have been a major recipient of Foundation 
implementation funding over the years. For exam|)le-, for fiscal 
years 1974 and 1975, EDC curriculums received approximately 
21 percent of the .Fo'unda't ion's total implementation fvrhds* 
Because before fiscal yeat 1974 the Foundation did not have 
dita readily avairable for total ptecollege implementation 
funding ' (Foundation-supported and non-Foundation-suppor ted 
*, currdculums) , we reviewed t-he implementation funding for^tthe 
Foundation's 53 major curriculums for which data was avail- 
able* Seven of those -53 curriculums were EDC developed* 
EDC curriculums have received 25, 21, 22, 19,^16, 15, and' 
-22 percent (an average of 19 percent) of the total Foundations, 
implementation funding for the 53 curriculums for fis'cal years 
1967-73, respec>-ively. ..." 

ACKNOWLEDGMENT OF SUPPORT 

Since as early as 1955, .the* Foundation has requested an 
acknowledgment of its .financial support to be included on all 
published mater ials developed with its funds* Not until late 
1972 or early 1973 did the Foundation also require a disclaimer 
,statem€^nt. The Foundation included in its October 1973 grant 
administration manual the following, requirement of an acknowledg- 
ment of support statement and a disclaimer statement • 

"An acknowledgment of NSF [National Science Founda- , 
tion] support must be 'made in connection with the 
publication of any material based on, or developed 
> under, a project supported- by NSF, along the, follow- 

ing lines: ' This , (material ) was prepared with the 
support of National Science. Foundation Grant 

No. .' Any opinions/^ findings, conclusions, or * 

recommendations cpressed in such a publication 
are those of the author(s) and do not , necessarily 
reflect the views of NSF, Any curricular materials 
' * * * must contain a statement 5l(2inq those lines, 
* * *" 

According tp^ Foundation officials, a disclaimer statement 
was required to eliminate any implioation that the Foundation 
endorses products just because it supported theit preparation. 
Also, the Foundation did not want to project the image that it 
was controlling the design and development of educational 
materials, V • 

"The MACOS materials did not include a disclaimer state- 
ment because the Foundation did not formally adopt this proce- 
dure until after signing the MACOS publication's contract in 
J U X y 1 7 u . 



According to Foundation officials, they do not "View all 
educatioS materials developed with/oundation support before 
their publication and have no routine procedure to insure that 
aud^ materials include a Foundation' disclaimer statement. In 
I?^d! tSeJ relj on the grantees and contractors to comply 
with the requirements. * - 

Although Foundation officials have said that the Fo^"^^" 
tion does not endorse the science education ^^^erials developed 
through its support, the Foundation's provision of iJPlemen- 
Stio? finds for suih materials may be viewed by "'ambers of 
the Sucatidnal conununity and others as an endorsement of the 
materials. • ' - 

CONCLUSIONS ' . . 

The Foundation does not have definitive ^^"^ iter ia .concern- • 
ing how long and to what extent precollege <^^^^^^fJ^lJ^f^^_.^^ 
eligible fo? implementation funding. Even P^°l %f ^J^^^' « 
cialiy marketed'^for a number of years are 
for Foundation implementation funding. /^•^^^^^^P^?^.,"^^^^ 
implementation grants were awarded about 5 Y^ars after the 
materials had-^ been commercially available. Due to the in 
novative -nature and expense to schools adopting some 
JoundaUon-supported materials, the Foundation may want to 
p?oJi^ implementation funds after the materials are com- 
mercially available. We believe, however , such a P^ctice 
Soes behefirthe commercial publisher of the ""aterials and 
maypro?ide an unfair advantage over publishers of competing 
products. 

The Fouhdajtion receives unsolicited proposals for imple- - 
menting major c^urriculum and course developments at the pre 
coneS leiel and subjects them to a ?°">P«titive review proc- 
ess. Foundation officials place considerable weight their 
orofessional opinions in finally determining which proposals, 
to fund ?he Foundation does not always include written docu- 
mentaJf^n In its files giving the reasons for the differences 
between the program staffs' judgment and the panelists rat.- 
between the^pr g documentation should be included for , 

^a?^ proposal, giving the reasons for support or nonsupport. 

The Foundation, which requires a disclaimer statement 
on all published materials developed with Fpundatipn support, 
has no Routine procedure to insure that, su^h a statement is ^ 
in fact included. The Foundation should insure that all 
Foundation-supported materials include a disclaimer state- 
men?f particularly because the Foundation provides implements- 
tibn funds for materials. 
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RECOMMENDATIONS 

We recommend that the Director of the Foundation: 

— Obtain the publishing industry's; views about any impact 
that tlje award determination and funding practices of 
'the precollege instructional improvement implementation 
program may have on the supported educational mate- 
rials and their publishers, and if necessary^ experi- 
ment with program revisions to minimize any unfair 
advantage. • ^ 

. —Require the Foundatibn program staff to document in 
the files the reasons for support or nonsupport of 
implementation proposals. 

'--Establish procedures to insure that the Foundation's 
acknowledgment of support and disclaimer statement is 
included on all published sciejice education materials 
which it has funded. 

AGENCY C OMMENTS 

— - ■ ■ — — * ^ 

The Foundation agreed with our recommendations and stated 
that actions are being taken to implement them. 
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CHAPTER & 

EDC'S ADMINISTRATION OF ROYALTY I^OME 



Educational materials developed with Foundation support 
may produce royal ties. 1/ The royalties are usually paid di- 
rectly to the grantee for a disposition to be determined by 
the Foundation r because it supported development of the ma- 
terials. Royalties returnecf by the grantees to the Founda- 
tion, are depositefJ in the U.S. Treasury. The Founda(tion may 
permit grantees to retain royalties to be used for purposes 
it approves. The use of these funds by the grantee ^ although 
approved by the Foundation ^ is not reviewed by the Congress' 
through the appropriation process. 

. The Foundation permitted EDC to retain royalti«[S from 
t^undation-funded projects. Our review of '^selected EDC roy- 
alty fund transactlQns indicated that the Foundation^ needs 
toXmore closely monitor fund activities to insure that all 
royalty income is correctly determined and used only for au- 
thorized purposes. , 

FOUNDATION ROYALTY IN.COME POLICY 

Befdre October. 1964 the Foundation did notth'ave a formal 
policy for\the disposition of royalty income derived from 
projects it\ supported; however r it generally placed specific 
provisions ih grants instructing grantees to retain any roy- 
alties iEor a disposition to be determined by i^.< In October 
1964 the Foundation adopted a formal policy requiring grant- 
ees to remit royalties received on Foundation-funded projects 
for deposit in the U.S. Treasury.' In February 1969 the Foun- 
dation changed it^ policy, to allow grantees to use royalty 
income to defray gr^^ant administration co^s with prdor Foun- 
dation approval. I^ January 1972^ the Foundatipn again re- 
vised its policy^^ tJjls time to' allow grantees to use royal- 
tiesy with prior Foundation Approval ^ to offset costs nor- 
mally chargeable to the^ grant and to cover reasonable ex- 
penses associated with administering the income-producing 
activity. The 1972 policy also provided that the Foundation 
may authorize the grantee ^^to retain grant income estimated 
to be less th.an $10^000 to ,be used* for science' or science 
education purposes.' However r any income exceeding $10^000^ 
or such lesser amount as may be specified in the gr.ant^ was 
to be remitted to, the Foundation unless approved by the 
Poflhdation for other usfes. \ 



1/A royalty is an ampunt usually based on a percentage of 
sales paid by commercial publishers to authors as consider- 
ation for the right to sell materials developed, by the au- 
'thors (.or' to which the authors otherwise have rl^ghts) . 
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. /The 1972 revisions, also provided that ail in^come and 
inte^.est remitted to the Founda'^tion of required -to be^ re- 
ported on by the grantees and contractors will be in turn 
rep'or^ted by the Foundation to the Office of Management and - 
^ddget (OMB) and to the Congress. 

The Foundation repor'ts. to JQMB for inclusion in the 
President's budget the income^ remitted to it by grantees and 
contractors. However, the Foundation does nbt report to OMB 
or the Congress the income it authorizes grantees and con- 
tractors to retain, and the total amount of such income is 
not readily de terirtinable from Foundation records. ^ In one 
documented examplfe, however, the Foundation had allowed^ EDC 
to use from royalties earned on Foundation-spnnnr ted projects 
about $1.4 million and retain about '^v': . i:or future au- 
thorized disposition. j • 

FOUNDATION ROYALTIES EARNED . ' 

\ ON EDC PROJECTS 

.\ * • \ . , 

\ Before April 1, 1970, EDC either , remitted royalties to 

\the Pouhdation or retained them for; disposition to be deter- 
mined by the Foundation. Such general handlirfg of royalty 
income was, consistent with Foundation practices or policies 
;a\t that time." Since April 1, 197-0:, the Foundation has au- 
^thorized EDO to retain ^li, royalty ihcome from ail 
Foundation-supported pr«>je5ts in a^combined •account . Founda- 
tion ^officials stated that\EDC is the only grantee permitted 
to^^ po62^ royalty income and \that the arrangement was consid- 
ered, the most practiQ^l one i^ecause of the number and magni- 
tude of Foundation-supppr ted ^curriculum materials EDC admin- 
istered. EDC-collected royaities, reported semiannually to 
the \ Foundation, are invested in interest-bpar ing accounts, 
cer t.if icates of deposit, and U.S. Treasury bills. 

\Per iodicaily, EDC submits a request to the Foundatil>A 
for using part of the pooled royal ty income with a' work .de- 
"Scrip^tiqn and a *budge*t"^"§timate . The Foundation reviews 
the submission and, when it concurs, authorizes EDC to use 
the royalty income for grant adgiinistration activities and 
other apprcjved purposes. This practice is consistent with* 
current Foundation income policy. / % 

' ' - * ' 

j Foundation royalty income activities at EDC from ' 
Ji^ly 1, 1960, through March 31, 1975*, are shown below. . 




Income : ' , ^ ' * 

Royalties received . 
Interest ■ ^ 

Expend i tur e s ( no te b ) : 

Returned^ to the Foundation 
Administrative costs ■ 
Other purposes 



a/ $3, 4^7 #484 
128^698 



$3,616,182. 



1,892,456 
1^85,527- 
313,178 



J: 



the Foundation 



3.29i;i61 

4 

$ '325,021 



a/$a92,102 attributable «to MACOS.^ * . 

b/slnce rbyaities are hfeld in one account and expenditures are 
ndt related to funding source, the MAQOS toyalties used for 
administration expenses or grant \activities or stxll owed 
the foundation are not identi£iab^.e . However, before • 
Wiil 1970, when funds were reinitted-^directly to- the 
Foundation, $5,340 of MACOS royalties were rpmitted^ 

EDC's administrative. costs includr.costs of those actir 
vities\ lelated to the. administration of terminated Foundation 
grantsnand the operation of its film lihfrary and distribution 
center ,\ "Film library functions include storing and cataloging 
films and handling requests for materials. Charges to the 
distribution center include selecting publishers, negotiating 
contracts, and /providing public information. ^- . -t 

" ' \ ' ' '■ 

♦ The Foundation authorized $313,178 of royalty income to 
fund the following grant activities. " . .> 



General 'grant 
. title 

PhyGieal 'science 
s tudy • coimni ttee 



Science education \. 



Developmental bi- 
ology films 

Fluid mechanics 
film program . 

Total ■ 



. Purpose of funds , 

Make tests, films^ and 
evaluations related to 
physics coyrses' developed 
by the conunittee 

Explore, and develop the 
use of- certain' science. ed- 
^ lication films 
• p* * . . 

Cover a cost overrun on the 
the grant 

-Publish a volume 'of- film - 
• notes \ ■^ 



Amount 'pf • 
/ funds 

/ $182,783 



,99,000 



27 ,'35 5 



. 4,040 



$^13-, 178 



ERIC 



(5. 

/MONITORING THE' EDC ROYALTY. POOL 



n general^ the Foundation's Grants and Contracts''«0£f ice^ 




reports, of income. The Audit Office is responsible for .au-/ 
diting grantee income earned on Poundd^tloh-supported projects 
and for examining, and reporting on practices and procedures ^ 
regarding income management /and use. Th^ last Foundation ^ 
audit at 6DC was through the peciod pnded March 1969. 

0ur ""limited review of selected rdyaity fund .transac.-' 
' tiohs atnd the uniqueneso of the royalty pool to the Founda- 
tion indicate that the Foundation should n^ore frequently 
and closely review the pooi; We noted the ^ following ques^ ' 
tionable transaction and expense allocation issue. 

Disposiftioji of royalty income • ' - 

In. the spring of 1970, EDC made a^ film entitled ••The 
, Eskimo: Fight for Life** for CBS Television; Inc. The rilmJ 
was an. edited version of the Netsilik Eskimo .films" funded *(^,^ 
. byVthe Foundation' and the National Film ^Board of Canada.^ ' 
' 'Their^ contracts, speqified that either party would- pay the* ; <> 
other ro percent of royalties received from direct televi- 
sion sale of films. 'In June 197.0, CBS paid EDC .,$68vl04 . f or . 
editing, .titli^ng, and other proc" ction cost« and $10,000 in - 
' royartiea^^ a.iietwork broadcast of "Fighy^t^/Life. ••* EDC 
remitted a* $a,;O.QO royalty payment to the Nation^J^ilm .Board 
and credited $9.^000 to the Foundation royaltj^ acg^Hint. In May 
1971, JBS paid EDC* an additional- $10 ,000 in rpyaltps f^r 
1 another sh9wing. of the film. EDO again remitted $1,,.000 tp' 
^ the^National Film Board and credited $9:,000 to the/Founda-. 
r tion royalty acc'ount. ^ ' / • • 

] ' * In^ September .15)71, EDC proposed to 'pu rehash, the non- 
'exclusive f ights to the stock Xo'^tagie 1/ us'ed- in the pro- 
gram,' and in about* Dpfeember 1^971 or January 1972, the Foun- 

- dation iffformally approve^ the purchase. , In July 1972 EDC 
sent th.e. Foundation a check' for ?9,180 of corporate funds to 
purchase^ the rights^ The Foundation returned the check for 
deposit'^ to the royalty account. tKus malting these funds 



1/Def ir^ed"^* as all of the filih. taken on a project that Irould 
"*.be used -to produce sUb^eq.uent f-ilms. \ • 
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available to EDC fo-: future authorized "^t^' /^l .f j^^ ' 
the. Foundation apparently acted contrary Jto P.S.C 484, ^ 
which .-requires that the gross amount of all _sums "^eceivea 
bv OVS. -officers and agents tor Government use be depositea 
tl the b-.S. Treasury. Since , the $^,180 wis J" f ^men^^^ ^ 
a riaht- owned^ by the Foundation and did ndt constitute roy 
-llti- income under a grant, the Foundation jacted -improp^^cly . 
-S'JeJCrning ?he moneys. Even if the income coulo be con- - 
sidered a royalty due the United States, lit would appear 
tiat since sucS^oyalty would be In the hjnds,^ of the agency, 
it wiuld have, to be deposited in the U.S. ^Treasury and coula 
not.be used to augment the agency's appropriation. • ^ 

EDC believed that the purchase of the j stockXfootage 
aa^e it undisputed ownership of the royalto^es; m June 
19^2, EDC transferred $19,865, representing CBS >^oyjlty 
paUents of $18,000 and applicable if from the Eou^^^ 

dation. royalty account to corporate funds. EDC had notified 
the FoundLioh of the royalties earned from' showing, W film, 
.a^d eSc-s removal of the royalties from the f'o""d^ation_ ac- 
count bv statements of royalty account activity subm.i\ttea to 
?he Foundation? The Gran'ts and Contracts Office rev^eweo- 
the income statements but apparently dip not question the 
wUhdrawa! of the royalties. Because the royalty income was 
earned before EDC's purchase of stock ^ °f ^?^/^?S^^Ao^^l,3. 
believe that the Foundation is entitled to the $18,000 plus, 
accrued interest derived from the "Fight for Life" proouc- 
. tion* * • ^ 

EDC officials, acknowledged the retroactive nature of 
this transaction and indicated that it would have been more 
apirdpriate to purchase stock footage, rights before use. We / 
- we?e advised that, although the $19,^65 was spent Primarily 
^o promote the "Fight for Life" film, MACOS indirectly bene- 
fited, from reference to 'the course in "Fight for Life mar- 
keting materials. • ^ . 

Sharin g administration costs 

Thk Foundation and EDC share royalties from the physi- . 
cal science study committee project based on their percent- 
ages of investment in the project (the Foundation invested 
86.8 percent and. EDC invested from private sources^ 13 .2 per 
cent).! However, EDC does not share m project administra- 
tion costs. 

An EDC official advised us that the determination of 
the ambunt.of Foundation and EDC investment was negotiated 
on the basis of a complex set of facts ana, legal i^^ues 
EDC claimed that 13.2 percent was the minimum to which it 
was entitled. However, this settlement maae no mention ot 
expenses relating to future royalties. 
' ' ' . 47. 
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Sihce^ both the Foundation and EDC share in royalty 
income andr benefit from the royalty pool expenditures sup- 
porting the project^ we believe that EDC , should assume a 
proportionate share of the project's administration costs. 
Such cpsts^ are not readily identifiable; however, based on 
•the intome-shar ing ratio of 86*8 to 13.2, we estimated that 
EDC's share of administration^cdsts would amount to about 
$49,000.,- * . J 

CONCLUSIQNS ... 

The Foundation's current practices for disposition of 
income resulting from its /grants and contracts permit the 
use of funds that are not J reported to the .Congress for con- 
sideration in appropriating funds to the Foundation. The 
significance of the amoun/t of such unds has not been ^^deter- 
mined, but the one grantee considered in our review, EDC, 
has been authorized to use substantial amounts of funds. 

The uniqueness of t/he royalty pool arrangement with EDC 
and the questionable transactions ideiitified in our limited 
review, of pool activities indicate that the Foundation needs 
^^to more closely monitor and review royalty pool transactions. 

RECOMMENDATIONS' ' ' * . , f ^ 

We recommend that the Director of the Foundation: 

— Providfe for a ^ comprehensive review of^EDC's royalty 
• fund transactions, including .the matters identified 

in this] report, and make necessary Adjustments. 

—Evaluate the results of the review and consider the 
need for more frequent audits of the EDC royalty fund. 

Y ^ ^ " ' ^ 

-^Insure that /edC income reports are closely examined 
to help monitor the royalty fund. 

AGENCY COMMENTS \ 1 \ 

The Foundatian. agreed with bur recommendations and stated 
that actions are '^2ing /taken to implement them. 

MATTER FOR CONSIDe!rATi{)N BY THE CONGRESS 

The Congress may/ wish to direct the Foundation's Direc- 
tor to determine ti\e^ significance of amounts of income re- 
spiting from grant!s\ and contracts that the Foundation is au- 
thorizing its gran't^es and contractors to use. If such 
amounts are signif'icant,, the Congress may wish to require 
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the Director to report the receipt ind expenditure of roy- 
alty income retained by the grantees and cp^tractors. This 
information would allow the Congress to consider these funds 
in determining the Foundation's annual appropriation.- 



CH APTER 7" 
V SCOPE OF REVIE W 
Otir review was primarily directe'd to 

— ^identify the i?oundation 's policies and procedures for 
developing, evaluating , and implementing science edu- 
cation projects administered by its Division of Pre- 
College Education in Science; 

— identi^fy the Foundation's policies and proqedures for 
•'disposition of -royalty income that may result from 
its science education projects.; and 

—determine the application of those policies and pro- 
cedures to the MACOS project. 

In conducting the review, we: 

— Interviewed officials of the Foundation; EDC; CDA; 
the National Endowment for the Humanities; and the 
Department of Health, Education, and Welfare and 
personnel who formally evaluated MACOS. 

7-Ascer tained the authority and functional responsibili- - 
ties of the Foundation's precollege science educa- 
tional division and the National Endowmeiit for the 
Humanities. 

--Reviewed the Foundation's project records for MACOS 
and similar EDC records,. 

— Determined the extent of competition for developing 
and' marketing MACOS by reviewing the process for 
^awarding grants for MACOS 'devel9pment and the process 
*for obtaining a publisher, including the royalty rate 
determination. } ► 

— Examined the {4AC0S professional services *agreement 
between EDC and CDA, the Found atipn's role in the 
agreement, and the general services provided under 
the agreement. 

— Reviewed the transactions under the professional 
^services agreement for matcbing funds through the 
National Endowment ton the lumanities. 
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—Examined the Foundation's process for awarding 
educational material implementation tupds 4t the 
•precollege level to ascertain if proposals for 
funding were competitively evaluated. 

— Reviewed the peer, review evaluations of MACOS 
proposals and. the objective testing of MACOS 
during development. 

—Inquired into the development and application of the 
Department of * Health/ Education , aftd Welfare ' s pol- 
icy and regulations for protecting humem subjects 
^ an'd specifically considered the adoption of the polr- 
icy by the Foundation and any applipation to MACOS. 

— Identified the Foundation's methods jCojc disclaiming 
any endorsement of science education, products it 
funds. 

- —Generally inquired into EDO's practl^en of pooling 
royalties resultina from Foundation-suppprt^d 
education materials and the general dlsppsition of 
.such royalties, with emphasis on MACOS royalties. 

—Gathered background information on the corporate 
- history of EDO and CDA and conducted a general in- 
quiry into their relationships 'to the FoAindation. 
'(See app. I.) 

Our review was conducted at, the Foundation in Washing- 
ton, D.C.. , and at EDO offices in Cambridge and Newton, 
Massachusetts, and CDA's office in Washington, p.C. 
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RELATIONSHIPS' BETWEEN 
THE FOUNDATION, EDC, AND CPA 
NATIONAL SCIENCE FOUNDATION 

The Foundation is an independent agency of the Federal 
Government established by the National Science Foundation 
Act of 1950. The Foundation's functions include: 

— Supporting basic scientific research to strengthen 
research potential and science education programs 
at all levels in the various scientific disciplines, 
including the social sciences, by providing financial 
assistance, such as grants, loans, scholarships, and 
fellowships. ^ ' ^ 

— Supporting applied research at nonprofit institutions, 
.and, when, authorized by the President, at other organ- 
izations When relevant to national problems involving 
the public interest, 

'l ■ - • ^ 

— Fostering interchange of scientific information among 
scientists in the United States and other countries* 

— Evaluating the s^atus andjneeds of the various 
sciences. ! 

/The Foundation receives appropriated funds to conduct 
its activities. Fiscal year 15-75 appropriations were about 
$768.2 million. 

The Director of the Foundation is appointed by the 
President and confirmed by the Senatfe. The Director, in 
promoting scientific research and educa-ion, is assisted by, 
among others, a National Science Board, which consists of 
24 members who are appointed by the President Snd confirmed 
' by the Sisnate and who"" are to be eminent in the fields of 
basic, medical, 6r social sciences; engineering; agricul- 
ture; education; research management; or public affairs . 

EDUCATION DEVELOPMENT CENTER, INC. 

Educational Services, Incorporated, of Newton, Massachu- 
setts, established in August 1958 to administer a new high 
school physics curri^uljum developed 'by a Massachusetts In- 
stitute of Technology professor and his colleagues, received 
Foundation financial support for the curriculum project. 
National emphasis on improving educational techniques^ and 
reforming curriculums led to expansion of Educational Serv- 
ices' activities. It developed curriculums and trained 
teachers and preceded a regional educational laboratory 
system .established by the Elementary and Secondary Education 
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Act of 1965. The act provided for- the Department of Health, 
Education, and Welfare's Commissioner of Education to award 
grants for, among other things, establishing curriculum re- 
search' centers, statewide educatipnal planning , measuring ' 
educational achievement, and improving teacher education. / 

In August 1966, the Institute for Educational Innovation 
.was established under the act's title IV', which called for a 
national' program of educational laboratories. The Office of 
Education was the only sponsor of the Institute, whose oper- 
ational intent was to address honschool factors in education, 
such as parental involvement in school activities at the com-r 
munity level. Educational Services, on the other hand, dealt 
with curriculum reform. In January 19'67, Educational Service* 
and the Institute merged to form Education Development Cen- 
ter, Inq. 

» 

EDC, organized to effect cur riculum 'reform, has essen- 
tially the same charter as its predecessor organizations. 
Xt is a nonprofit membership corporation with, ho stock- 
issuing authority. Control of. the corporation is vested in- 
the board of trustees. 

Support for the organization's projects comes from both 
public and private sources; however, from its inception (as 
Educational Services) to September 30, 1974, *y.S. Governmojit 
agencies had provided about $86.5 million (84 percent) of 
tptal EDC funding of $102.7 million.* The, Foundation had pro- 
vided about $33.8 million (about one-third) of total EDC' 
funding. . * 

CHRRICULOM development ASSOCIATES, INC. 

CDA, incorporated in January 1970 to develop and dis- ^ 
semiriate curriculum programs, is an expansion of its predeces- 
sor organization, Wirtz-Botel Associates. According to CDA ^ 
officials, the corporation was established partly because of I 
Wirtz-Botel Associates' dissatisfaction with the publishers / 
it had selected for their curriculum programs. 

' In. addition to publishing and distributing MACOS, CDA 
publishes and disseminates a new math curriculum it developed. 
A CDA-developed elementary reading program is disseminated by 
another publisher under a royalty agreement. 

EDC, CDA, AN'D FOUNDATION RELATI ONSHIP S 

T,he Committee, requested us to provide a history of E^DC 
and. CDA, including the relationships between the two organi- 
zations and their principal .officials, and to determine ^the , 
relationships between EDC, CDA, and the Foundation. 
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We identified key officials at EDC and CDA (fx.om the 
d2^|:es these organizations were established to May 2, 1975) 
and their predecessor organizations (Educational Services, 
the Institute, Wir tz-Botel) . We reviewed EDC's anriual re- 
ports for i967-74 and its personnel files. ^We revised 
CDA's listings of officers (inception to May 5, 1975) and 
stockholders as of December 31, 1974. We also compared the 
listing of key EDC and CDA officials with Foundation^ employ- 
ment termination records dating back to about 1970 and its 
employment records of personnel on board as of May 31, 1975. > 

. One CDA official was involved with pilot te;sting\ MACOS; 
however, th^is involvement was as an employee of a local 
school system and not as an EDC employee. . - I 

Another par t-time' CDA consultant was formerly employed 
by Egf? in various capacities fox; over 4 years. She was a 
consu^ltant on the social studies program from March to ljune 
19(65, a socia'r studies staff member from June to July 1965 
and September 1965 to, August 1966, and the" Director of ^Man 
and Other^ Animals" from September 1966 to August 1969. \^he 
has been employed by CDA from, July 1973 to the present as a 
part-time, consultant to work on teacher-training workshops' 
and the MACOS revision. / 

A third individual joined EDC in March 1.969^ to help 
develop a MACOS dissemination strategy and other activities 
related to preparing MACOS for commercial publication. * 
Although the individual left full-time employment at EDC °in 
July 1969 to join a private foundation, he continued on a 
part-time basis until April 1970. From November 1969 throtigh 
April 1970, he received $250 per month for work related to ' 
MAQOS dissemination." ' \ 



A CDA Official told ufe that this same individual, al-r 
though never employed by ^DA, did assist CDA in developing 
an overall^ dissemination jmodel for its programs, including 
MACOS, during the period January through June 1970. In Au- / 
gust 1970, CDA, in appreciation of his help, issued him 250 
shares of stock in the company.' He advised, us that the stock 
was promptly returned because of the potential conflict of 
interest with his employment at the private foundation-. « 

One former Educations^ Services official was previously 
a member of the t^pundation' s National Science Board, but the 
periods of employment were not concurrents In addition, 
three former ED^Educational Services officials have served 
as Foundation consultants. 
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APPENDIX II 

NATIONAL SCIENCE. FOUNDATION 

WASHINGTON. D.C 20550 
SEPT. 15, 1975 



' Mr* Gregory J. Ahart ... * 

Director, Manpower and • 

Welfare Division 
U* S. General Accounting ^Office 
Washington, D. C.^ 20548 

Dear Mr. Ahaft: * , . 

The draft GAO report on Administration of the Science E4ucatlon Project, 
" "Man: A Course of -Study" (MACOS) has been reviewe4- In order to resolv? 
the problems outlined 'In my letter to you of August 13/ 1^75, representsr 
tlves of GAO and NSF staffs have been working together over the past few 
weeks and fi^ive^resolved, to their mutual satisfaction, Ht^ft of the apparent 
» 

problems. 

• Generally, we concur in the recommendations made in report; they 
parallel those made by the NSF Pre-College Science Curriculum Review ^ 
Team either in its published report or in the admini.^r/itive recommenu.- 
tions made separately. The policy, administrative, or procedural change, 
needed to implement these recommendations are currently being developed 

- by the appropriate NSF staff. » 

There are certain conclusions and suggestions cont«lne4 in the body of 
the report, however, that have not been completely resolved during the 
• discussions between the GAO and NSF staff representatives. 

Statements are made in a number, of places which imply the need for direcf 
involvement of NSF in the operations of its grantees and contractors. 
For example, in Chapter 4 it is stated that NSI has no ^P^^^^J^ ^^"^^'j 
criteria for grantees to consider in selecting « publisher. In Chapter 5 
. it is suggested that as a minimum the Foundation should ensure ajl 

FouiLtioh-supported materials include a disclaimer statement. We recog- 
nlS the nee'd to. improve and tighten up the administration of Foundation 
curriculum development programs, but believe NSF should be primarily 
involved in establishing overall policy for guidance of grantees and 
contractors to effectively carry out such programs. 

In Chaoter-^3, "Evaluation of MACOS," there are'S==e aspects which I believe 
require comm;nt. The chapter ties together two^^unrelated subjects - 
"Evaluation" and "Protection of Human Subjects. 

The "Protection of Human Subjects" was provided for in the scope of your 
review. While it is thus appropriate to the report, placing it in 
juxtaposition with "Evaluation" makes it appear to be an issue related 
sdlely to evaluation. It is more properly an isaue for the chapter on 
c development . 
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The discussion ce.ntered around the Department of Health, Education and 
Wf^if are's guidel:lLnes on protection of human subjects. As pointed out, the 
NSF has voluntarily adopted the policies of DHEW, although not In the case 
of educatloj^ar activities. This Is In accord with DHEW thinking. NSF was 
advised bjl^pproprlate DHEW officials that the guidelines were nevet meant 
to apply to curriculum -development. Tliere are instances where the policies 
(as contrasted to the guidelines) are at)plied to educational activities * 
by the Offilce of Education^ and th^^ National Institute of Education but not 
usually to curriculum development grants. 

We will undertake to review the need for proceduWs to protect human subjects 
under NSF educational programs, f ~~ 

The GAD report concludes th^t a-, comprehensive evaluation of MACOS was ^ 
performed. They question the attribution of pre- to post-test gains to 
MACOS because of Imitation in EDO's evaluation design;" further they 
question the adeqjjftacy b£ EDO's reporting of test .results. We understand ^ 
these ccncluslorlB were reached after analysis of only oi^e of 78 findings 
in the evaluation ifeport. 

We have had 'many discussions with the GAG representatives and have not 
^een able to agree oh' the validity pf GAO's conclusions, particuliarly 
since their opinion of the meanj.ng of the specific finding is at variance 
with the evaluator's and NSF*s opinion. There are also differences of 
opinion a^ to the .significance of a formative evaluation as opposed to a 
summative evaluation. There ate also some questions as to the adequacy 
of reporting. 

Since the effectiveness of educational evaluation is being considered .in 
a separate GAG review, we believe it appropriate that further <>comment be 
withheld at tfhis time. ^ . ' 

In Ohapter 4, "Obtaining a Odramercial Publisher for MAOOS," closer monitoring 
and a comprehensive review of EDO royaltiesP^are discussed. The report 
states that the low royalty rate (3%) used.,f or MAOOS printed material 
might have improved the, competition for a MAOOS publisher. We do hot 
believe the low royalty rate would have favorably influenced publishers 
to reconsider, since their primary reasons for rejection of MAOOS were 
not related to royalties. No evidence has been offered which demonstrates 
that the royalty rate was a significant factor during the three-year publisher 
selection process for MAOOS. 
> 

Also in Ohapter 4 tihe discussion of the professional services contract 
between EDO and ODA covers the use of funds for purpbses not related to 
MAOOS but, related to the National Endowment for the Humanities. Accordingly, 
this item was not discussed by the GAG and NSF staffs and will not be 
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coinmented--on. Concerning the question whether NSF should have reviewed 
and approved the contract between CDA and EDC on the basis it was a 
subcontract coming within the scope of, the NSF publication "Grants for * 
Education in Science," we believe this is a matter. of interpretation. 
However, we believe NSF., in fact, handled the matter correctly. 

In Chapter 6, "Administration of Royalty Income ," Iquestions concerning two 
transactions in the EDC royalty account are'raised. These are presently 
under review by NSF staff. ' \ ^ 

I am pleased to note that our previous concerns about the tone and substance 
of the draft report have been largely rectified and the areas of difference 
have been- reduced to the ones above.. 

Xs noted above, the G AO review has identified certain administrative areas 
of the NSF Pre-College Cujjriculum Program which the. Foundation also 
recognizes as needing to be strengthened.. Action is being taken to do 
this. We appreciate the opportunity to comment on this draft report. 

Sincerely yours. 




fel A. Snow 
"Director, Office of Planningv 
r * and Resources Hanagement 
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MmVlSKM OF SCIENTIFIC PERSOpEL 
And EPUCATION (note d): 
Associate Director . 
• Henry W. Riecken 
Bo wen C. Dees 
• <^ssistant Director 
Bowen C. Dees 
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Teniire of office 
FromV 



To 



Mar . 

Aug . 

Aug. 



1964\ Jan. 
1963 \. Mar. 

1959 A/g. 




a/As Executive Associate Director, Dr. Levin performed the . 
fjanctions of Deputy Director. ' * 

b/Effective July 1975,. title changed to. Assistant Director 
•"" for Science Education. * < 

c/Effective October 1969, the Office of Assistant Director 
for Education was created. In addition to the duties as- 
signed to the newly established position, the Assistant 
Director for Education assumed theVf unctions previously 
assigned to the Associate Director (Education). 

d/Effective January 1965, the "Division of Scientific Person- 
~ nel and Education was abolished and replacedl by three 

divisions, one' of which was the Division of Pre-College 
' Education. in Science, reporting to the Associate Director 

(Education) . . 
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